Nicholas Carlini
Nicholas Carlini is a research scientist at Anthropic who studies how machine learning systems can be attacked. In 2016 he and David Wagner publish three attack algorithms that succeed on both distilled and undistilled neural networks, and their 2018 ICML paper on obfuscated gradients wins a best paper award. His 2020 and 2021 work shows that language models memorize some of their training text. The 2024 paper Stealing Part of a Production Language Model earns an ICML best paper award. He works at Google Brain from 2018 to 2023 and at DeepMind from 2023, then moves to Anthropic in 2025.
Timeline
-
2016
The Carlini & Wagner attack
In 2016 he and David Wagner publish three attack algorithms that succeed on both distilled and undistilled neural networks.
-
2018
A PhD under David Wagner
He finishes a UC Berkeley PhD under David Wagner in 2018. His B.A. in computer science and mathematics is also from Berkeley.
-
2018
An ICML 2018 best paper award
At ICML 2018 he, Anish Athalye, and David Wagner present Obfuscated Gradients Give a False Sense of Security, which wins a best paper award.
-
2018–2023
A research scientist at Google Brain
He works as a research scientist at Google Brain from 2018 to 2023.
-
2020–2021
Language models that memorize text
He shows in 2020 that language models memorize some of their training text. In 2021 he presents a USENIX Security paper, written with ten co-authors, that measures the privacy of large language models.
-
2024
A model-stealing attack
His paper Stealing Part of a Production Language Model wins an ICML 2024 best paper award. The attack extracts precise, nontrivial information from black-box production language models.
-
2025
Moving from DeepMind to Anthropic
He leaves Google DeepMind, where he worked from 2023, and joins Anthropic in 2025 to continue his adversarial machine learning research.
-
As of October 2026
Research on language model misuse
At Anthropic he works as a research scientist, studying what bad things you could do with, or do to, language models.
Sources and credits
Photo: National Institute of Standards and Technology, via Wikimedia Commons. License: Public domain. Source: commons.wikimedia.org. Cropped to a square and resized. Ask us to remove or replace this photo.
