PeopleofSuperintelligence

Researcher

Kaiyue Wen

Kaiyue Wen is a Stanford PhD student who studies how large language models are pretrained, from optimizers to attention design. A graduate of Tsinghua University's Yao pilot class, he coauthored a 2022 paper on Sharpness-Aware Minimization with Tengyu Ma and Zhiyuan Li before joining Stanford in 2024. There, advised by Tengyu Ma and Percy Liang, he explained warmup-stable-decay learning rates through a river valley loss landscape in October 2024 and benchmarked ten optimizers in September 2025. The gated attention paper he coauthored in May 2025 won a NeurIPS 2025 Best Paper Award, and its output gating is used in the Qwen3-Next models. He is a PhD student in computer science at Stanford, where most of his pretraining research runs through the open-source project Marin.

Timeline

  1. 2022

    Papers on sharpness and skills

    In 2022, he coauthors "How Does Sharpness-Aware Minimization Minimize Sharpness?" with Tengyu Ma and Zhiyuan Li, a paper that pins down the exact sharpness notion that SAM regularizes. He also coauthors the EMNLP 2022 paper "Finding Skill Neurons in Pre-trained Transformers via Prompt Tuning".

  2. 2024

    PhD at Stanford

    He is admitted to the computer science PhD program at Stanford University in autumn 2024, after graduating from Tsinghua University as a member of Yao's pilot class. At Stanford he is advised by Tengyu Ma and Percy Liang.

  3. October 2024

    The river valley loss landscape

    In October 2024, he posts "Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective" as first author, with Zhiyuan Li, Jason Wang, David Hall, Percy Liang and Tengyu Ma. The paper conjectures that pretraining loss resembles a deep valley with a river at its bottom, which explains the loss curve of the WSD schedule.

  4. May 2025

    Gated attention for language models

    In May 2025, he is a coauthor of "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free", which finds that a head-specific sigmoid gate after scaled dot-product attention consistently improves performance. The same output gating is used in the Qwen3-Next models.

  5. September 2025

    Benchmarking pretraining optimizers

    In September 2025, he posts "Fantastic Pretraining Optimizers and Where to Find Them" with David Hall, Tengyu Ma and Percy Liang, a study of ten optimizers across four model scales. It finds that the speedup of many proposed optimizers over well-tuned baselines falls with model size, to only 1.1x at 1.2B parameters.

  6. November 2025

    A NeurIPS Best Paper Award

    In November 2025, the gated attention paper he coauthored receives a NeurIPS 2025 Best Paper Award, one of four best papers chosen from the Main Track and the Datasets & Benchmark Track.

  7. As of October 2026

    Pretraining research with Marin

    He is a PhD student in computer science at Stanford, where most of his pretraining research runs through the open-source project Marin. His work covers optimizers, architectures and hyperparameter scaling for large language models.

Career

Education

  1. From 2024Stanford UniversityPhD
  2. Tsinghua University

From the biography.

  1. 20% of people at companies who shape AI also hold a university role, led by Google and Stanford UniversityIncludes Stanford University. We estimate that 539 of the 6,766 people with a known current organization (8%) hold both a university role and a company role, including 20% of people at companies and 13% of people at universities.
  2. Google had the most people shaping AI every year from 2014 to 2025Includes Stanford University. Our estimates of which organizations employed the most people shaping AI each year from 1955 to 2025, tracing the shift from university labs to companies.
  3. More people shaping AI studied at Massachusetts Institute of Technology than at any other institutionIncludes Stanford University. We estimate where the people shaping AI studied and where their careers led. Massachusetts Institute of Technology leads with 467 former students, including 291 recorded doctorates.
  4. The company share of AI researchers’ top papers was 31% in 2020–2024 and 8% in 1970–1974Includes Stanford University. We estimate where the people shaping AI worked when their most-cited papers appeared: the company share was 31% in 2020–2024 and 8% in 1970–1974, and Google led all organizations in 2020–2024.

Colleagues

People named in this biography, and people with dated positions at the same organization in the same years.

Sources and credits

Photo: We have no published photo of Kaiyue Wen yet. Send us a free-licensed photo.

  1. [2211.05729] How Does Sharpness-Aware Minimization Minimize Sharpness? arxiv.org. Checked 21 October 2026.
  2. Kaiyue Wen whenwen.github.io. Checked 21 October 2026.
  3. Kaiyue Wen's Profile | Stanford Profiles profiles.stanford.edu. Checked 21 October 2026.
  4. [2410.05192] Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective arxiv.org. Checked 21 October 2026.
  5. [2505.06708] Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free arxiv.org. Checked 21 October 2026.
  6. Announcing the NeurIPS 2025 Best Paper Awards – NeurIPS Blog blog.neurips.cc. Checked 21 October 2026.
  7. [2509.02046] Fantastic Pretraining Optimizers and Where to Find Them arxiv.org. Checked 21 October 2026.
  8. NeurIPS 2025 Awards neurips.cc. Checked 21 October 2026.