Optimizing federated learning with communication reduction and synchronization control

Abstract

Machine Learning (ML) models have brought breakthroughs in natural language processing, speech recognition, computer vision, and many other fields. These ML models have gone deeper in recent years, requiring large-scale datasets to produce more accurate models. However, the traditional datacenter approach often necessitates uploading the input data to the centralized server for model training. With the ever-increasing concerns about data privacy, such an approach is facing tremendous challenges when privacy is paramount.

Federated Learning (FL) becomes a promising alternative ML paradigm, aiming to leverage user data for model training in a privacy-preserving way. In the FL setting, the edge devices collaboratively train a shared model under the orchestration of a central server while keeping data locally. Yet, the training performance is often poor due to various reasons, such as huge communication overhead, system heterogeneity, and data heterogeneity. The cross-device FL training usually involves a massive number of resource-weak edge devices connected to intermittent networks, exhibiting a vastly heterogeneous environment. The unbalanced and non-independent and identically distributed (Non-IID) property of data distribution also leads to an accuracy degradation. This thesis highlights the primary challenges in FL systems and proposes three efficient optimization strategies that are specialized for the cross-device federated learning setting.

We first propose a structure-based communication reduction algorithm, named FedSCR, to reduce the number of parameters transferred via the network without compromising accuracy. We study the parameter update pattern under the Non-IID setting and discover the update tendency over channels and filters. Herein, FedSCR effectively quantifies the magnitude of model updates by aggregating them over channels and filters, and then removes the insignificant updates by comparing the aggregated values with a threshold. We further introduce an adaptive FedSCR that dynamically adjusts the threshold for different clients based on their weight divergence, the magnitude of local updates, and data distributions.

Then, we address the straggler problem by presenting KAFL, a fast K asynchronous FL framework to control the model synchronization under data and resource heterogeneity. KAFL includes two synchronization algorithms, named K-FedAsync and M-step-FedAsync, which make a trade-off between system efficiency and model accuracy. Compared to the fully asynchronous manner, they help the server reduce training bias and obtain a better direction toward the global optima as it collects information from at least K clients or M different model updates. We also introduce a weighted aggregation method to determine the aggregation weight for each client according to its weight deviation and contribution frequency.

Finally, we extend the FL training from deep neural networks (DNNs) to graph neural networks (GNNs) and propose an EmbC-FGNN framework. EmbC-FGNN enables node embedding communication among edge clients with privacy guarantees. It contains an embedding server (ES) to maintain the shared embedding vectors, and allows the edge clients to expand their local subgraphs without revealing local node features and graph topology. To minimize the communication costs of the ES, EmbC-FGNN advocates a periodical embedding synchronization strategy to reduce communication frequency, and it further utilizes K-FedAsync and M-step-FedAsync algorithms to accelerate convergence speed.