Pavel Izmailov
Contact: pi390@nyu.edu, Twitter
I am a Researcher at xAI. I am primarily interested in reasoning and AI for math and science.
Starting in Fall 2025, I will be joining NYU as an Assistant Professor in the Tandon CSE department, and Courant CS department by courtesy. I am also a member of the NYU CILVR Group.
Previously, I worked on reasoning and superintelligent AI alignment at OpenAI.
My research interests are broadly in understanding how deep neural networks work. I am excited about a broad array of topics in core machine learning, including:- • Improving reasoning and problem-solving in AI
- • Interpretability of deep learning models, including both large language models and computer vision models
- • AI for scientific discovery
- • Out-of-distribution generalization and robustness of large-scale models
- • Technical AI alignment
- • Probabilistic deep learning, uncertainty estimation and Bayesian methods
Our recent work on weak-to-strong generalization was covered by a WIRED, MIT Technology Review and others. Our work on Bayesian model selection was recognized with an Outstanding Paper Award 🏆 at ICML 2022!
Links
- [Home, Bio, Publications, Talks, CV, GitHub, Google Scholar, Semantic Scholar]
Selected Papers
-
*Equal first authorship. Full list of papers available here.
-
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
2023
[PDF, ArXiv, OpenAI blog, Code] [WIRED, TechCrunch, MIT Technology Review, IEEE Spectrum] -
FlexiViT: one model for all patch sizes
Conference on Computer Vision and Pattern Recognition (CVPR), 2023
[PDF, ArXiv, Code] -
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
International Conference on Learning Representations (ICLR), 2023 🌟 Spotlight Presentation
[PDF, ArXiv, Code] -
On Feature Learning in the Presence of Spurious Correlations
Neural Information Processing Systems (NeurIPS), 2022
[PDF, ArXiv, Code] -
Bayesian Model Selection, the Marginal Likelihood, and Generalization
International Conference on Machine Learning (ICML), 2022
🏆 Outstanding Paper Award, 📢 Long Talk (Oral)
[PDF, ArXiv, Code] -
Dangers of Bayesian Model Averaging under Covariate Shift
Neural Information Processing Systems (NeurIPS), 2021
[PDF, ArXiv, Poster, Code] -
What Are Bayesian Neural Network Posteriors Really Like?
International Conference on Machine Learning (ICML), 2021
📢 Long Talk (Oral)
[PDF, ArXiv, Code, HMC samples, Poster, NeurIPS competition] -
Why Normalizing Flows Fail to Detect Out-of-Distribution Data
Neural Information Processing Systems (NeurIPS), 2020
[PDF, ArXiv, Code] -
Bayesian Deep Learning and a Probabilistic Perspective of Generalization
Neural Information Processing Systems (NeurIPS), 2020
[PDF, ArXiv, Code] -
A Simple Baseline for Bayesian Uncertainty in Deep Learning
Neural Information Processing Systems (NeurIPS), 2019
[PDF, ArXiv, Code, Poster, Video] -
Averaging Weights Leads to Wider Optima and Better Generalization
Uncertainty in Artificial Intelligence (UAI), 2018
📢 Oral Presentation
[PDF, ArXiv, Code, Poster, Slides, PyTorch Blogpost, Towards Data Science Blogpost, fast.ai Blogpost] -
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
Neural Information Processing Systems (NeurIPS), 2018
🌟 Spotlight Presentation
[PDF, ArXiv, Code, Poster, Slides, Video, Blogpost]