AI systems are increasingly embedded in society. Their interactions with people are more frequent and complex—and the outcomes of those interactions become more unpredictable. Nika Haghtalab believes we need to build AI in a way that directly addresses this uncertainty.
“If you don’t know how [an AI system] was built foundationally, you don’t know what kind of impacts or weaknesses they’re going to have, or what solutions they may come up with,” says Haghtalab.“That’s the issue I’m addressing when we think about mathematical foundations of AI.”
Nika Haghtalab is a 2023 AI2050 Early Career Fellow and Assistant Professor in the Department of Electrical Engineering and Computer Sciences at UC Berkeley. Haghtalab and her work have been recognized with multiple awards, including a 2024 Sloan Fellowship, a 2023 Google Research Scholar award, and a 2022 NSF CAREER award.
Haghtalab’s research focuses on the theoretical foundations of machine learning, particularly learning and decision-making algorithms that operate within social or economic contexts. Her AI2050 project builds AI models that perform equitably across multifaceted social considerations and remain robust under unpredictable circumstances. This project addresses Hard Problem #2, solving issues on AI robustness and performance in safety-critical contexts.
The AI2050 initiative gratefully acknowledges Fayth Tan for assistance in producing this community perspective.
My work offers one perspective of using mathematical grounding, in particular using theoretical computer science, statistics, and economics. What I mean [by foundational] is that I should be able to predict what kind of outcomes I’m going to get based on the information and type of environment that I have.
I’ll use construction as an example: when setting up a building, you examine the ground conditions and the specifications of the building you plan to build—if you are in a sandy versus rocky bed or building a high-rise versus a single-family home, you’re going to require different things. Similarly, we should plan and account for what kinds of environments our AI systems will operate in and what we want them to achieve. Perhaps I’m in an environment where people actively try to break into the system or manipulate its behavior, or perhaps I’m in an environment [where] the tool that I’m developing is going to be used by experts [for] a well-defined and specific purpose.
I can and should account for these environmental factors and capabilities when I build the model. [They all] tell me something: in a building, it might tell me how deep my foundation has to be. In a learning system, it might tell me how I should optimize a model, and what kind of data and at what scale is needed.
Some of the most widely used models now are large language models. They are developed in several phases, where you first throw everything you have at it. You get a huge model that’s powerful, but doesn’t necessarily act in any way that you want. Then, you try to shape its behavior, piece by piece through, what’s referred to as post-training, to act more desirably in situations you expect the model to face. These methods do improve the usability of the models but they don’t come with mathematical guarantees.
An example of this is ChatGPT, where the underlying model is very, very big, and its behavior is shaped through many rounds of post-training. When it’s interacting with humans, it’s supposed to act in a safe way. But the second such a model is released, people start trying to get it to do things that it wasn’t [originally] expected to be asked to do.
Both the study and the development of AI [lack] guarantees about what will happen once these systems interact with people. With more mathematical and scientific guarantees, we understand not only the capabilities and limitations of our system in isolation, but also [how they perform] in the wild, when interacting with humans.
The framework is built on a multi-objective learning framework. When you’re designing an AI system, you might have many different objectives—for example, if you’re developing a risk predictor for heart disease, one objective is to have good performance on different segments of the population. I don’t want this risk predictor only to do well on a 25-35 year-old white population, I also want it to do well on a 50-75 year-old Asian population.
Typically, having more objectives is [about] trying to capture more robust and reliable model performance. [But] if we have multiple objectives, how do we go about optimizing for them? It has to address many technical challenges — how do you define objectives? How do you identify them? What happens if the objectives aren’t fully aligned with each other? Those are the challenges of trying to achieve multiple objectives all at once.
Objectives are literally what people want from a system. But when it comes to human objectives, it’s hard for us to give an exact list of prioritized outcomes, let alone assign values to them.
In [the case of the] heart disease predictor, someone might want it to be very accurate. Another person might want it to [avoid] costly errors for the hospital system. Maybe someone only cares about their own health outcomes, so they’re willing to have errors as long as they’re going to live as long and happily as possible. Human factors play a role even in the definition of objectives.
This is going to be a challenge of multi-objective learning, because we’re dealing with concepts that are hard to express for humans. But eliciting human feedback about how well the system does during its development will be one way to approach defining and refining these objectives.
I think more people are talking about how AI is made because of how people’s data is being used. A lot of today’s technology runs on data — data [that] we made available because we didn’t think that it had value to us, both monetarily or even in terms of privacy. We made it available at a scale of billions of people. I’d like to see more discourse on that data’s provenance — the impact individuals and groups have had [in its creation], and the control they can have over their own data.
Absolutely, AI isn’t something grown out of the ground and foraged! We haven’t lost control of how AI systems are being developed, far from that. It’s important to realize that we can make decisions to have better AI systems, and that there are many things that shape the future of AI. The development of AI itself is shaped by social and strategic forces. Let’s make sure that [those forces are] going to benefit people.