Demystifying Data Poisoning Attacks in Distributed Learning as a Service

Abstract

Data Poisoning is a dominating threat in the distributed learning-as-a-service API, where the mediator has limited control over the distributed client contributing to the joint model. Through an in-depth characterization of data poisoning risks in federated learning, this paper presents a comprehensive study towards demystifying data poisoning attacks from three perspectives. First, we formally define the targeted dirty-label data poisoning attack, which aims to cause the trained global model to only misclassify the input from a specific victim class with a designated malicious behavior. Then, we demonstrate theoretical statistical robustness in the eigenvalues of the covariance in the gradient update shared from the client to server when under the data poisoning attack. Second, we study the impact of attack timing and identify the most detrimental attack entry point during the federated training. Last, we examine several existing defenses against data poisoning in addition to the robust statistic detection. Through formal analysis and extensive empirical evidence, we investigate under what conditions the statistical robustness of data poisoning can serve as the forensic evidence for attack mitigation in federated-learning-as-a-service, at what attack timing the attack is most detrimental, and how the attack reacts in the presence of the existing defenses.