Large language models (LLMs) have a tendency to memorize and output data—even when that data might be your social security number.
Anjalie Field wants to safeguard against that outcome by establishing a “gold standard” for privacy in AI, developing tools to protect privacy in safety-critical fields such as healthcare, law and social services. She speaks to the nuances of working at the intersection of technology and social science, navigating the context-dependent nature of privacy.
“Privacy is hard to define,” says Field. “Think about what you posted on Facebook in high school. You might regret certain posts now, but at the time, you didn’t care about exposing that information.”
Anjalie Field is a 2024 AI2050 Early Career Fellow and Assistant Professor in the Computer Science Department, Whiting School of Engineering at Johns Hopkins University focused on the social impacts and ethical development of AI systems, including collaborations in child protective services and police accountability records. Her work tackles Hard Problem #2, solving safety and security challenges in AI to preserve public trust and safety.
The AI2050 initiative gratefully acknowledges Fayth Tan for assistance in producing this community perspective.
This powerful new technology comes with a lot of potential privacy risks. Models might reproduce the data that they ingest, which means data must be handled carefully in sensitive contexts. This affects who can be allowed to use a model, or what a model can be used for, as they can be used in ways that violate or compromise people’s privacy.
Privacy also often comes with a trade-off between research that we might want to do, or values that we care about. For example, one way to enhance responsible AI is to ensure that models are independently audited so that there is transparency and oversight in their development. However, giving access to auditors comes with privacy risks.
There is often a trade-off between utility and privacy as well. You could protect privacy simply by sharing nothing, but that restricts what you can do. If you don’t have any data, you can’t act on it. Increasing privacy protections might come at the cost of how useful that data may be to train a model, for example. The question of how we define metrics to measure that trade-off is part of what we’re working on.
When it comes to LLMs specifically, models trained on sensitive data are liable to memorize and output that information. This includes personally identifying information such as your social security number or banking and credit card details, or legally protected information that could be used to target people, such as health data. That means we have to rule out use cases where exposing that information would be inappropriate—a patient-facing model trained on clinical data including a case study of a highly specific genetic condition might resurface that case study verbatim in response to another patient’s general health query. LLMs are also good at making connections—they may be able to infer sensitive attributes even if information is redacted, such as being able to infer the identities of individuals from an anonymized dataset.
With use cases like AI for mental health, interactions can be deeply personal as well. While a technology provider having a lot of information about you is not new per se—Google had all of your search history before OpenAI had your ChatGPT logs—the type of information disclosed, whether it’s shared in a response to another user, leaked, or extracted by an attacker, could be much more sensitive.
The first approach focuses on protecting data. Historically, text data has been protected through de-identification, by removing information like names or social security numbers. In an LLM era, I don’t think this protection is strong enough. We’ve been exploring the use of synthetic text as a substitute for real text when training AI systems. LLMs are good at generating data that appears fluent, but is not totally accurate. We could leverage that to produce synthetic data that doesn’t contain any real private information, but is realistic enough to be useful.
Instead of protecting the data, the other approach involves changing what information a model learns from the data. Could we ensure a model only learns specific information? Or control what it outputs as an answer? Essentially, we’re asking if a model can be taught to respect privacy.
Ultimately, we want to give people the ability to choose the right tool for the right circumstance. For example, if a hospital shares data with a small research team to build a healthcare model for internal use, simple de-identification might be acceptable. In contrast, if the hospital wanted to release a model for other hospitals, higher standards of privacy might be needed. To ensure those standards are met, the hospital could accept the trade-off of having a less accurate model. We want to create a go-to toolkit to support effective trade-offs, maximizing utility given differing levels of privacy needs.
In general, I think everyone should think about their own data more! More specifically, pay attention to what technology providers are doing with your data. Based on how your data is used, what do you or do you not want to feed into technology products? How might it change the way you use these tools?
If you’re interacting with an LLM through a model provider’s interface, like how many people use ChatGPT, they often train on your data. OpenAI’s Terms of Service explicitly states that, depending on your settings and service version, they train on user queries, so it’s not a secret, but I think that the ramifications aren’t immediately clear to everyone. I can’t say I always read all of the Terms of Service, but at least look up what it says about data use. If you type something into an LLM, and you’re not careful about data terms, you’ve got to be prepared that one day, it might come right back out as a response to someone else’s query.