James Martens

James Martens is a machine learning researcher whose work centers on second-order optimization for neural networks. He presented Hessian-free optimization for deep auto-encoders at ICML in 2010 from the University of Toronto, and in 2015 he proposed Kronecker-factored Approximate Curvature (K-FAC) with Roger Grosse. After a computer science PhD under Geoffrey Hinton and Richard Zemel, he joined Google DeepMind in September 2016. The 2018 paper "The Mechanics of n-Player Differentiable Games," written with David Balduzzi and others, won an ICML Best Paper Runner Up award. Later work includes Deep Kernel Shaping in 2021 and a 2023 paper on training deep transformers without shortcuts.

Now at: Google DeepMind

Timeline

2010

Hessian-free optimization at ICML

In 2010, he presents "Deep learning via Hessian-free optimization" at ICML from the University of Toronto. The 2nd-order method trains deep auto-encoders without pre-training and beats results Hinton and Salakhutdinov reported in 2006.
2011

Two recurrent network papers

In 2011, he publishes "Learning Recurrent Neural Networks with Hessian-Free Optimization" with Ilya Sutskever at ICML. At the same conference, he coauthors "Generating Text with Recurrent Neural Networks" with Sutskever and Geoffrey Hinton.
2013

Initialization and momentum in training

In 2013, he coauthors "On the importance of initialization and momentum in deep learning" with Ilya Sutskever, George Dahl and Geoffrey Hinton. The paper shows that stochastic gradient descent with momentum, a well-designed random initialization and a slowly increasing momentum schedule can train deep and recurrent networks to levels once reached only with Hessian-Free optimization.
2015

Introducing K-FAC

In 2015, he and Roger Grosse propose Kronecker-factored Approximate Curvature (K-FAC), an efficient way to approximate natural gradient descent in neural networks. The paper appears at the 32nd International Conference on Machine Learning.
September 2016

Joining Google DeepMind

After a computer science PhD at the University of Toronto under Geoffrey Hinton and Richard Zemel, he joins Google DeepMind in September 2016.
2018

An ICML runner-up paper

In 2018, ICML names "The Mechanics of n-Player Differentiable Games," by David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls and Thore Graepel, a Best Paper Runner Up. The paper decomposes the second-order dynamics of games into potential-game and Hamiltonian-game parts, which motivates the Symplectic Gradient Adjustment algorithm.
2020

Natural gradient in JMLR

In 2020, the Journal of Machine Learning Research publishes his "New Insights and Perspectives on the Natural Gradient Method." The paper shows how natural gradient descent can be viewed as a 2nd-order method in which the Fisher information matrix substitutes for the Hessian.
2021–2023

Training deep networks without shortcuts

In 2021, he leads a paper that introduces Deep Kernel Shaping, which enables SGD training of residual networks without normalization layers on ImageNet and CIFAR-10. In 2023, he coauthors "Deep Transformers without Shortcuts" with Bobby He, Yee Whye Teh and others, which designs approaches for training deep transformers without normalisation.