Cyber Defense eMagazine December Edition for 2021

validation study by Abdalla et al. (2020) [Mohamed Abdalla, Moustafa Abdalla, Graeme Hirst, and
Frank Rudzicz. 2020. Exploring the privacy-preserving properties of word embeddings:
Algorithmic validation study. J Med Internet Res.].
There are four types of disclosure concerns when it comes to protecting data privacy:
●
●
●
●
Identity disclosure: identifying an individual.
Attribute disclosure: identifying an individual’s ethnicity, religion, physical attributes, etc.
Group attribute disclosure: e.g., is a particular group more likely to have cancer?
Membership disclosure: e.g., is this person part of a pharmaceutical trial?
Not all attributes are the same with regards to increasing the risk of these disclosures. Within
conversations with chatbots, users might reveal direct identifiers (e.g., full names, exact addresses,
phone numbers, credit card numbers) and quasi-identifiers (e.g., religion, origin, gender, etc.). When
combining quasi-identifiers together, the risk of re-identifying an individual grows exponentially.
The ScatterLab incident mentioned above is an example of identity and possibly attribute disclosure,
though one major issue was actually membership disclosure through identity disclosure. These
disclosure types were caused by the leak of direct identifiers and perhaps of quasi-identifiers as well.
Preventing Identity, Attribute, and Membership Disclosures
There are a few solutions for dealing with training data memorization within chatbots. One is differentially
private gradient descent (DPGD), which was used in Carlini et al.’s 2019 paper. DPGD adds noise to the
ML model training process. The original idea behind differential privacy is to be able to make
generalizations about a population without the risk of disclosing any specific individual’s unique
information. The goal of adding differential privacy to an algorithm, like a chatbot model, is that if you run
the algorithm on two datasets differing by a single entry, then the likelihood of getting a different set of
possible outputs is negligible. DPGD provides mathematical guarantees that rare information is not being
memorized by a machine learning model, though often at the expense of model utility.
Another solution is highly accurate redaction or de-identification, which means removing the direct
identifiers and quasi-identifiers within your training data (e.g., location, names, telephone numbers, etc.).
There’s a lot you can gather from a conversation’s context without the need for identifiable information.
Finally, another option is using synthetic personal data generation. This method allows for replacement
of direct and quasi-identifiers in a very natural way, so a chatbot’s training data matches the style of the
language model’s pre-training dataset, which prevents downstream model accuracy loss. It also has the
additional benefit that, if any personally identifiable information is missed, it’s very difficult to tell what the
original data was from the synthetic data. Targeted synthetic data generation changes the paradigm of
disclosure risk versus data utility.
If ScatterLab had used either one of these three methods to protect the privacy of their users, they would
have prevented violating training data privacy, as well as input and output privacy. Their story inspires
Cyber Defense eMagazine – December 2021 Edition 106
Copyright © 2021, Cyber Defense Magazine. All rights reserved worldwide.