Sam Shleifer

Timeline

1. March 2019

   ULMFit and backtranslation

   In March 2019, he posts "Low Resource Text Classification with ULMFit and Backtranslation" as sole author. A ULMFit model fine-tuned on 50 IMDB examples plus 500 backtranslated examples reaches 80.6% accuracy, an 8.1% improvement over the baseline without augmentation.

2. October 2020

   The Transformers library paper

   He is one of 22 authors of "Transformers: State-of-the-Art Natural Language Processing", which describes the open-source Transformers library and its Transformer architectures under a unified API. The paper appears in the EMNLP 2020 system demonstrations and receives an Honorable Demonstration Paper award.

3. October 2020

   Distilling summarization models

   At Hugging Face, he and Alexander M. Rush post "Pre-trained Summarization Distillation" in October 2020, comparing knowledge distillation, pseudo-labeling and "shrink and fine-tune" on Pegasus and BART. Shrink and fine-tune outperforms the other two on CNN/DailyMail, and checkpoints of several sizes are released through Hugging Face transformers. His stated interests on Hugging Face include sequence-to-sequence models, summarization of long documents and fine-tuning methods.

4. October 2021

   NormFormer and 8-bit optimizers

   In October 2021, he posts NormFormer with Jason Weston and Myle Ott, a transformer architecture that adds three normalization operations to each layer. He also co-authors "8-bit Optimizers via Block-wise Quantization" with Tim Dettmers, Mike Lewis and Luke Zettlemoyer, which develops the first optimizers that use 8-bit statistics while keeping the performance of 32-bit optimizer states.

5. December 2021

   Mixture-of-experts language models

   In December 2021, he is one of 24 authors of "Efficient Large Scale Language Modeling with Mixtures of Experts", an empirical study of how autoregressive mixture-of-experts models scale against dense models. At modest training budgets the mixture-of-experts models match dense models with roughly 4 times less compute, and the largest, at 1.1T parameters, outperforms a compute-equivalent dense model.

6. May 2022

   The OPT language models

   In May 2022, he is one of 19 authors of "OPT: Open Pre-trained Transformer Language Models", work he does while at Meta AI. OPT is a suite of decoder-only transformers from 125M to 175B parameters that the authors aim to share with researchers.

7. April 2023

   The PyTorch FSDP paper

   In April 2023, he is one of 18 authors of "PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel", which introduces PyTorch Fully Sharded Data Parallel (FSDP) as an industry-grade solution for large model training.

8. Since February 2025

   Joining Thinking Machines Lab

   In February 2025, Mira Murati's Thinking Machines Lab announces 29 employees, and he is named among the former OpenAI research scientists on the team. He is a member of technical staff on the founding team, after earlier work as an AI research engineer at Character.AI.