FedEFsz: Fair Cross-Silo Federated Learning System With Error-Bounded Lossy Compression
Zhaorui Zhanga b, Sheng Dic d, Benben Liue f g h, Zhuoran Jii j, Guanpeng Lik l, Xiaoyi Lum n, Amelie Chi Zhouo p, Khalid Ayed Alharthiq r, Jiannong Caoa b
- aDepartment of Computing, Hong Kong Polytechnic University, Hung Hom, Hong Kong
- bDepartment of Computing, The Hong Kong Polytechnic University, Hong Kong
- cArgonne National Laboratory, Lemont, IL, USA
- dArgonne National Laboratory, USA
- eLSCM R&
- fD Center, University of Hong Kong, Pokfulam, Hong Kong
- gLSCM R&
- hD Center, The University of Hong Kong, Hong Kong
- iSchool of Cyber Science and Technology, Shandong University, Qingdao, China
- jShandong University, China
- kUniversity of Iowa, Iowa City, IA, USA
- lUniversity of Iowa, USA
- mUniversity of California, Merced, CA, USA
- nUniversity of California, Merced, USA
- o
Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.
Abstract
Cross-Silo federated learning systems have been identified as an efficient approach to scaling DNN training across geographically-distributed data silos to preserve the privacy of the training data. Communication efficiency and fairness are two major issues that need to be both satisfied when federated learning systems are deployed in practice. Simultaneously guaranteeing both of them, however, is exceptionally difficult because simply combining communication reduction and fairness optimization approaches often causes non-converged training or drastic accuracy degradation. To bridge this gap, we proposeFedEFsz. On the one hand, it integrates the state-of-the-art error-bounded lossy compressor SZ3 into cross-silo federated learning systems to significantly reduce communication traffic during the training. On the other hand, it achieves a high fairness (i.e., rather consistent model accuracy and performance across different clients) through a carefully designed heuristic algorithm that can tune the error-bound of SZ3 for different clients during the training. Extensive experimental results based on a GPU cluster with 65 GPU cards show thatFedEFszimproves the fairness across different benchmarks by up to$60.88\%$and meanwhile reduces the communication traffic by up to$315\times$.