Cyber Defense eMagazine December Edition for 2021

4 Pillars of Privacy-Preserving AI
Understanding the privacy challenges that chabots face requires, first and foremost, a general
understanding of what the privacy challenges are for machine learning systems in general. There are
four pillars to privacy-preserving AI:
1) Training data privacy: making sure that you can’t reconstruct sensitive or personal information
within the training data,
2) Input privacy: privacy of the individual whose data you’re inferring upon,
3) Model weights privacy: privacy of the model of a particular corporation, institution, or individual
who created it. This is about IP protection, but also training data privacy, since it is possible to
determine information about the training data from model weight updates,
4) Output privacy: also about protecting the privacy of the individual whose data you’re inferring
upon.
By collecting private conversations with identifiable individuals and training their models on them,
ScatterLab first violated (2) input privacy, then (1) training data privacy, and possibly (4) output privacy.
Training Data Privacy
Much of research and development these days focuses on training data privacy, in part because of how
likely deep learning models are to memorize training data, with the potential of spewing it out in production
to unknown parties. The secret sharer: Evaluating and testing unintended memorization in neural
networks [Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The
secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th
USENIX Security Symposium, pages 267–284, Santa Clara, CA. USENIX Association.] by Carlini et
al. (2019) is a pivotal paper discussing the problem. They placed a fake social security number into the
Penn Treebank dataset as a canary and then trained a character language model on the dataset. They
then measured the perplexity of various sequences of numbers and found that the model was less
surprised to see the sequences of numbers that made up the canary; i.e., the language model had
recorded that it was more likely to encounter the canary rather than other random numbers given the
training data. This is a problem because it shows that the language model memorized the secret.
Another paper titled Extracting training data from large language models by Carlini at al. (2020)
demonstrates how GPT-2 was actually memorizing data from the pre-training dataset. [Nicholas Carlini,
Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam
Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2020. Extracting training data from large
language models. arXiv preprint arXiv:2012.07805.] It had memorized addresses, names, and other
information that could be considered sensitive had the data not been publically available. It is important
to keep in mind that these very models will be memorizing that same kind of information from chatbot
training data. The paper showed that an extra large GPT-2 model already started memorizing information
after seeing only 33 examples.
Privacy issues have also been raised about training non-contextual word embeddings on data containing
sensitive information in Exploring the privacy-preserving properties of word embeddings: Algorithmic
Cyber Defense eMagazine – December 2021 Edition 105
Copyright © 2021, Cyber Defense Magazine. All rights reserved worldwide.