Liyuan Liu

Timeline

1. 2016–2021

   PhD research at Illinois

   From 2016 to 2021, he is a research assistant in the Data Mining Group at the University of Illinois at Urbana-Champaign, working toward a PhD in computer science under Jiawei Han. A B.Eng. in computer science from the University of Science and Technology of China, where Linli Xu advised his undergraduate work, comes first.

2. 2018

   Language models for sequence labeling

   In 2018, the AAAI paper Empower Sequence Labeling with Task-Aware Neural Language Model appears, which he coauthors with Jingbo Shang, Xiang Ren, Frank Xu, Huan Gui, Jian Peng and Jiawei Han. Its framework adds character-aware language models to sequence labeling, and training on the CoNLL03 named-entity task completes in about 6 hours on a single GPU. His EMNLP 2018 paper on language model pruning for sequence labeling is selected as an oral.

3. 2019–2020

   The RAdam optimizer

   In 2019, he coauthors On the Variance of the Adaptive Learning Rate and Beyond with Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao and Jiawei Han, and the paper appears at ICLR 2020. It finds that the adaptive learning rate has a problematically large variance early in training, shows that warmup works as a variance reduction technique, and proposes RAdam, a variant of Adam that rectifies the variance.

4. 2020

   Admin for stable Transformers

   From January 2020 to May 2020, he works part-time in the Deep Learning Group at Microsoft Research Redmond. In 2020, he coauthors Understanding the Difficulty of Training Transformers with Xiaodong Liu, Jianfeng Gao, Weizhu Chen and Jiawei Han for EMNLP 2020; it finds that heavy dependence on a layer's residual branch amplifies small parameter changes and proposes Admin, an adaptive model initialization that stabilizes early training.

5. February 2022–October 2025

   Researcher at Microsoft Research

   In February 2022, he joins the Deep Learning Group at Microsoft Research in Washington as a senior researcher. In February 2025, he moves up to principal researcher in the same group.

6. 2023

   ReinMax for discrete gradients

   In 2023, he coauthors Bridging Discrete and Backpropagation: Straight-Through and Beyond with Chengyu Dong, Xiaodong Liu, Bin Yu and Jianfeng Gao. The paper shows that the Straight-Through heuristic works as a first-order approximation of the gradient and proposes ReinMax, which reaches second-order accuracy by integrating Heun's method. NeurIPS 2023 presents it as an oral.

7. 2024

   FastGen and an ICLR honor

   In 2024, the ICLR paper Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs, which he coauthors with Suyu Ge, Yunan Zhang, Minjia Zhang, Jiawei Han and Jianfeng Gao, receives an Honorable Mention for the Outstanding Paper Award. Its FastGen method profiles attention modules to build an adaptive KV cache without fine-tuning or re-training, potentially cutting memory use by half.

8. Since October 2025

   Joining Thinking Machines Lab

   Since October 2025, he is a member of technical staff at Thinking Machines Lab. His research examines the mechanisms behind common machine learning heuristics, such as learning rate warmup, the Straight-Through estimator, Transformer initialization and the off-policy mismatch between sampler and learner in reinforcement learning frameworks.