Abstract
We consider federated learning in tiered communication networks. Our networkmodel consists of a set of silos, each holding a vertical partition of thedata. Each silo contains a hub and a set of clients, with the silo's verticaldata shard partitioned horizontally across its clients. We propose TieredDecentralized Coordinate Descent (TDCD), a communication-efficientdecentralized training algorithm for such two-tiered networks. The clients ineach silo perform multiple local gradient steps before sharing updates withtheir hub to reduce communication overhead. Each hub adjusts its coordinates byaveraging its workers' updates, and then hubs exchange intermediate updateswith one another. We present a theoretical analysis of our algorithm and showthe dependence of the convergence rate on the number of vertical partitions andthe number of local updates. We further validate our approach empirically viasimulation-based experiments using a variety of datasets and objectives.