FedHe: Heterogeneous Models and Communication-Efficient Federated Learning

Abstract

Federated learning (FL) is able to manage edge devices to cooperatively train a model while maintaining the training data local and private. One common assumption in FL is that all edge devices have similar capabilities and share the same machine learning model in training, for example, identical neural network architecture. However, the computation and store capability of different devices may not be the same. Moreover, reducing communication overheads can improve the training efficiency but it is also a difficult problem in the FL environment. In this paper, we propose a novel FL method, called FedHe, inspired by a core idea from knowledge distillation, which can train with heterogeneous models, handle asynchronous training processes, and reduce communication overheads. Our analysis and experimental results demonstrate that the performance of our proposed method is better than the state-of-the-art algorithms in terms of communication overheads and model accuracy.