Jacob Hilton

Also known as Jacob Haim Hilton

Jacob Hilton is an AI alignment researcher and VP of research at the Alignment Research Center (ARC). After a 2016 PhD in set theory at the University of Leeds, he worked at OpenAI on reinforcement learning from human feedback, scaling laws and interpretability, co-authoring TruthfulQA, WebGPT and InstructGPT. With Leo Gao and John Schulman, he measured in 2022 how a gold-standard reward score changes as a model is optimized against an imperfect proxy reward model. In 2024, he spoke out about OpenAI's non-disparagement agreements and signed the "Right to Warn" letter from current and former employees of frontier AI companies. At ARC, he works toward mechanistic estimates of neural network behavior that outperform random sampling, and in 2026 he helped launch the ARC White-Box Estimation Challenge.

Timeline

2016

A set theory PhD at Leeds

He completes a PhD in set theory at the University of Leeds School of Mathematics in 2016 with the thesis "Combinatorics of countable ordinal topologies", supervised by John Kenneth Truss.
2019–2020

Understanding RL Vision at OpenAI

At OpenAI, he works on reinforcement learning and co-authors a 2019 paper with Karl Cobbe, Christopher Hesse and John Schulman on using procedural generation to benchmark reinforcement learning. In November 2020, he is the primary contributor to "Understanding RL Vision" in Distill, which applies interpretability techniques to an agent trained to play the video game CoinRun.
2021

TruthfulQA and WebGPT

With Stephanie Lin and Owain Evans, he publishes TruthfulQA in 2021, an 817-question benchmark on which the best model tested is truthful on 58% of questions against 94% for humans. In December 2021, he is second author of WebGPT, which fine-tunes GPT-3 to answer long-form questions by searching and navigating the web and then improves its answers with human feedback.
2022

InstructGPT and reward overoptimization

He co-authors the March 2022 InstructGPT paper, which fine-tunes GPT-3 with reinforcement learning from human feedback; human evaluators prefer outputs of the 1.3B-parameter InstructGPT model to those of the 175B GPT-3. In October 2022, he publishes "Scaling Laws for Reward Model Overoptimization" with Leo Gao and John Schulman, measuring how a gold-standard reward score changes as a model is optimized against a proxy reward model.
2023–2024

Theory research at ARC

In June 2023, he, Mark Xu and Paul Christiano announce a hiring round for the Alignment Research Center's Theory team, and by April 2024 he is one of ARC's full-time researchers. His June 2024 essay describes ARC's aim of combining mechanistic interpretability with formal verification through heuristic explanations. With Mark Xu, he presents ARC's September 2024 paper on backdoor defense, learnability and obfuscation as an analogy for deceptive alignment.
May 2024–April 2025

The Right to Warn letter

In May 2024, he says he signed a non-disparagement agreement on leaving OpenAI only to keep his vested equity, and OpenAI contacts him about lifting it. He signs "A Right to Warn about Advanced Artificial Intelligence" in June 2024 with current and former employees of frontier AI companies, a letter endorsed by Yoshua Bengio, Geoffrey Hinton and Stuart Russell. In April 2025, he is one of 12 former OpenAI employees named in a proposed amicus brief, filed by Lawrence Lessig in Elon Musk's lawsuit, that opposes OpenAI's planned conversion from a nonprofit to a for-profit corporation.
As of October 2026

VP of research at ARC

He is VP of research at the Alignment Research Center, a nonprofit working on the theoretical foundations of mechanistic interpretability. In May 2026, he presents ARC's paper on estimating the expected output of wide random MLPs, an early step toward mechanistic estimates that outperform random sampling for any trained neural network. ARC and AIcrowd launch the ARC White-Box Estimation Challenge in June 2026, which he announces with Paul Christiano and Wilson Wu, with a prize pool of at least $100,000.