Nicolas Papernot

Nicolas Papernot is an Associate Professor at the University of Toronto, in the Department of Electrical and Computer Engineering, the Department of Computer Science, and the Faculty of Law. He is also a faculty member at the Vector Institute where he holds a Canada CIFAR AI Chair, and a faculty affiliate at the Schwartz Reisman Institute. In addition, he is a Co-Director of the Canadian AI Safety Institute (CAISI) Research Program at CIFAR. His research interests span the security and privacy of machine learning. Some of his group’s recent projects include generative model collapse, cryptographic auditing of ML, private learning, proof-of-learning, and machine unlearning. Nicolas is an Alfred P. Sloan Research Fellow in Computer Science, a Member of the College of the Royal Society of Canada, and a Schmidt Sciences AI2050 Early Career Fellow. He is also a recipient of the McCharles Prize for Early Career Research Distinction and the ACM SIGSAC Outstanding Early-Career Researcher Award. His work on differentially private machine learning was awarded an outstanding paper at ICLR 2022 and a best paper at ICLR 2017. He co-created the IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) and co-chaired its first two editions in 2023 and 2024. He is currently the program co-chair of the IEEE Symposium on Security and Privacy (Oakland). Nicolas earned his Ph.D. at the Pennsylvania State University, working with Prof. Patrick McDaniel and supported by a Google PhD Fellowship.

AI2050 Project

Companies and countries training AI models increasingly face scrutiny from end users around the risks of deploying AI. Acknowledging that AI poses risks to society, it is a reasonable expectation that a regulatory body should produce technical specifications to curb the societal risks of AI models. In addition to the difficulties faced when defining properties AI systems should meet, auditing these properties remains out of reach at the scale needed to regulate AI internationally. Nicolas Papernot’s AI2050 project addresses this hard problem through a combination of AI and cryptographic advances that will lay the foundations for verifiable AI treaties that benefit all.

Project Artifacts

AI2050 Community Perspective — Nicolas Papernot (2025)

D. Glukhov, Z. Han, I. Shumailov, V. Papyan, N. Papernot. A False Sense of Safety: Unsafe Information Leakage in ‘Safe’ AI Responses. arXiv. 2024.

P. Maini, H. Jia, N. Papernot, A. Dziedzic. LLM Dataset Inference: Did you train on my dataset?. arXiv. 2024.