Just a few years ago, the possibilities of generative AI being used to commit digital espionage or create deepfakes convincing enough for a multi-million dollar scam were theoretical. Now, those hypotheticals are becoming real occurrences—increasingly so, as AI becomes more powerful and accessible. Eric Wong’s research confronts that stark reality.
“AI models can do things they couldn’t do before,” says Wong. “Those capabilities, combined with the fact that it’s in many more hands, vastly increases the scope of all the things that it could possibly do, both for good and for ill.”
Eric Wong is a 2024 AI2050 Early Career Fellow and Assistant Professor in the Department of Computer and Information Science at the University of Pennsylvania. His methods to test LLM safety have been used at companies developing frontier models, including Meta, Google, and Anthropic. A pioneer in “provably robust machine learning,” Wong’s AI2050 project develops safeguards to prevent the misuse of generative AI. His work addresses Hard Problem #2, which focuses on solutions to AI safety and security risks.
The AI2050 initiative gratefully acknowledges Fayth Tan for assistance in producing this community perspective.
One risk is misinformation or the misuse of information. Generative AI is frequently used for health information, for example. While knowledge gets into the hands of many more people, not everyone knows how to parse it. People may misdiagnose themselves due to a lack of understanding, or try out treatments based on faulty recommendations from generative AI.
Great capabilities also come with the great potential for danger. We’ve seen this in the past with technologies such as dynamite, which was initially used for mining but can also be used as a weapon. Similarly, AI coding agents, which usually assist with programming tasks, have been used to launch large-scale cyberattacks. People with malicious intent can also more easily access specialized knowledge such as how to build biological weapons.
We’ve been worried about these scenarios occurring in theory, but they’ve actually come to pass, and at a rate that’s faster than I would’ve hoped.
There are two problems depending on whether large language models (LLMs) are talking to a layperson, or if they’re talking to an expert. For the layperson, LLMs sound convincing since they use expert-like terms supported by hundreds of citations. However, the layperson has no idea how to verify the sources or review the LLM’s recommendations for accuracy.
In contrast, when we’ve worked with experts in healthcare, the problem is completely flipped. Clinicians reject LLM answers outright because what the LLM says doesn’t make sense in the language of medicine—they wouldn’t make the same recommendations, or consult the same studies. Since experts know how to navigate nuances and have a holistic view, the blunt tool provided by the LLM isn’t what they would apply in practice. It’s a problem of translation, of communicating the information from LLMs to experts in a way that makes sense to them.
We’ve focused on this problem of expert translation not only with doctors, but with astrophysicists, social scientists, and psychologists as well. We established what it means to align with an expert by finding common patterns in how experts think across all these settings, and measured how aligned LLM explanations are with an expert’s way of thinking. We used that information to guide LLMs towards answers that are more expert-aligned. Since laypeople trust LLM answers, ensuring LLMs produce expert-aligned answers is a much better starting point to prevent misinformation.
A year ago, we wrote a paper where we asked: as AI gets more powerful, in 10 or 20 years, what will the end game between attackers and defenders look like?
A malicious query—how can I build a bomb, for example—can be broken down into smaller series of questions, such as what the chemical reactions are or what temperatures are needed. By turning it into a string of basic science questions, the attackers’ plan can’t be detected on a single-query level. With coding agents, attackers can do this at scale, making a million attempts a week to bypass safeguards or exploit vulnerabilities.
Essentially, we envisioned that the attackers could cover their tracks by using new IP addresses or routing traffic through different parts of the world, and the final game is defenders trying to find a needle in a giant haystack. Defenders must sort through conversations routed through different channels all over the world, and find related sequences of questions that lead to a harmful goal.
This exact scenario actually happened after we published this paper—and not 10 years later, but rather in about 6 months.
We also showed what defenders could do to prevent such attacks. To tackle this deluge of information, we first sort and cluster queries into similar buckets. Based on each cluster, a defense algorithm determines if a cluster represents abnormal or malicious behavior. To find the needle in the haystack, we look for patterns of misuse. For example, if a user is testing the vulnerabilities of one company’s systems, they might just be an engineer for the company. However, if there’s a pattern of them doing the same thing at 100 companies, it’s much less likely to be benign.
I think theory is greatly underappreciated right now. We have so many big problems in safety and alignment, and the current strategy seems to be throwing many darts at the wall and hoping that something sticks. This strategy only works with GPUs that can run hundreds of thousands of experiments—maybe then one of them will stick and we’ll make advancements.
Theory has always been a way to figure out where to go without throwing all those darts at the wall. We’ve published papers on theoretical models of safety that have pointed us in the right direction, with insights from that theory successfully applied at scale. Besides efficiency, the inherent nature of theory guarantees results will be true decades from now—which isn’t necessarily the case for empirical results. Theory is a much longer-lasting science.
We need to educate people on how much they should trust LLMs. I see the importance even with my own parents—they’re asking LLMs about everything from daily life to medical issues.
People aren’t primed to look out for when an LLM is completely off the mark. You’ll see people on the news who ran into trouble because an LLM told them something incorrect, such as people who received misleading medical advice from chatbots and delayed necessary care as a result. We’ve got to make sure that people know how to make a judgment call on whether the information they’re being presented is true or fake.