Manipulating SGD with Data Ordering Attacks
Ilia Shumailova b, Zakhar Shumaylova, Dmitry Kazhdana b, Yiren Zhaoa b, Nicolas Papernota c, Murat A. Erdogdua c, Ross Andersona b
- aUniversity of Cambridge University of Cambridge University of Cambridge University of Cambridge University of Toronto & Vector Institute University of Toronto & Vector Institute University of Cambridge & University of Edinburgh
- bUniversity of Cambridge, Cambridge, United Kingdom
- cUniversity of Toronto, Toronto, Canada
Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.
Abstract
Machine learning is vulnerable to a wide variety of attacks. It is now well understood that by changing the underlying data distribution, an adversary can poison the model trained with it or introduce backdoors. In this paper we present a novel class of training-time attacks that require no changes to the underlying dataset or model architecture, but instead only change the order in which data are supplied to the model. In particular, we find that the attacker can either prevent the model from learning, or poison it to learn behaviours specified by the attacker. Furthermore, we find that even a single adversarially-ordered epoch can be enough to slow down model learning, or even to reset all of the learning progress. Indeed, the attacks presented here are not specific to the model or dataset, but rather target the stochastic nature of modern learning procedures. We extensively evaluate our attacks on computer vision and natural language benchmarks to find that the adversary can disrupt model training and even introduce backdoors.
TL;DRA novel class of training-time attacks that require no changes to the underlying dataset or model architecture, but instead only change the order in which data are supplied to the model, which finds that the attacker can either prevent the model from learning, or poison it to learn behaviours specified by the attacker. (Semantic Scholar)