The scale of data in modern genomics is staggering. Recent datasets such as the OpenGenome project contain nearly 9 trillion base pairs from genomes across the tree of life. Given the vast troves of training data available to today’s advanced AI models, a popular assumption is that AI must be on the verge of decoding the building blocks of life itself. But does biology demand more than increasingly massive machines to be truly understood?
Antonio Orvieto is a 2024 AI2050 Early Career Fellow and machine learning scientist developing foundation models for genomic research. With a background in optimization theory, which studies optimal strategies for decoding challenging data patterns, Orvieto has long been motivated to apply mathematical principles to complex biology. He recalls telling a mentor about his early ambitions to model biological pathways, even if there was limited data about them:
“He told me that, mathematically, what I was trying to do was completely impossible,” Orvieto says. “But that’s what learning is. In some way, we’re trying to solve the impossible with machine learning. That inspires me.”
His work addresses Hard Problem #1, which tackles critical scientific limitations that must be solved to enable new breakthroughs in AI. He’s interested less in the race to build bigger, more powerful tools, and toward the question of how and why they are built. Motivated by the many unknowns surrounding the promise of genomic models, his goal is to establish a rigorous, evidence-based path forward to ensure that AI delivers meaningful insights for biological research.
The AI2050 initiative gratefully acknowledges Fayth Tan for assistance in producing this community perspective.
Large language models for text are evaluated using benchmarks, or a well-established set of standardized tasks. Genomic models should also be evaluated against standardized, challenging benchmarks to demonstrate progress scientifically. However, the capabilities of new models are typically demonstrated using arbitrary tasks, like calculating the fitness of a certain set of genes. It is not clear if any given set of tasks has broader relevance to the field.
Every researcher is asking different questions based on their own interests—everyone is swimming in their own pond, so to speak. Researchers can show that their models have specific capabilities, or that their methods are impressive, but that is not the same as showing progress per se.
The next step should be to work with biologists to design benchmarks that act as clear metrics for model performance. For example, there’s decades of research into statistically modeling the structure of DNA. One of the projects we’re working on now tests if, as scale increases, genomic models return sequences with physical properties which actually align with what we know about the structure of DNA. Assessing if models return sequences with the right physical properties might be an example of a benchmark.
We might be left with very big models that can’t actually do much more than we can now. We wouldn’t know why we built the models that way either, beyond the fact that everyone else was also building big models with big data.
For example, pre-training models on the genome—including the genome in the big, diverse datasets used in the initial stage of training large AI models—has become more popular. But we don’t know if seeing so much data actually improves the performance of the model on biological questions. One of our Ph.D. students is evaluating how model size and training data impact downstream results, which is important as we need to make sure the way we’re building models is relevant to biological applications.
It’d be a shame to benchmark the wrong things when we could be developing a set of tasks that represents something difficult, that biologists care about, and that was not possible before. That, to me, would be evidence that biology can truly benefit from the current AI revolution.
When we ask and answer questions about text with LLMs, there’s a big overlap between the data, which is text on the internet, and the task that we want to do. There’s a lot of statistical associations between words, so you can learn the grammar—learning the data distribution allows you to predict the next token, the next word. In language, grammar implies something about semantics, about meaning.
When you pre-train on the genome, we’re learning correlation between nucleotides like we learn correlations between words in a sentence. This means models may be able to predict sequences very well—you can make a mutation in a sequence and ask if this is a sequence that might truly occur in biology.
Unlike language, however, there’s a bigger gap between the structure of DNA and its function. For example, does predicting the next nucleotide imply that you can predict whether the next region is a silencer or an enhancer? Given a sequence from a specific region of the genome, can you predict if the chromatin in that region is open? We can’t say for sure, and we don’t know if that’s possible yet. Sequence modeling alone may not be sufficient to infer function.
Models are not products. They’re not things that one buys at the supermarket. Models are also not forever. We will build them, of course, but the real product will be the path forward, about knowing how to go about building these models.
My goal is to understand how the construction of a model impacts its performance, and for the measure of that performance to be standardized across the field. My paper will be outdated after one year, but the point is to develop methodology that can be used and evolved by other people.
I’m very picky and careful about my claims, but ultimately, what I want is conclusive evidence that modern AI can be relevant for biology. I’d like to see evidence that clearly shows us we can do something now that was unthinkable before. I want to approach AI in biology with a clear, collective path forward, not just with random choices we’ve made along the way.