Blog
Training Dynamics of Bigram Induction
October 14, 2026
Two small simulations to frame how quickly and when bigram induction forms in a small transformer
The Dynamics of Learning
September 23, 2026
This post explores the hidden biases in loss minimization and frames learning in terms of dynamical systems.
A Toy Model of the Neural Tangent Kernel
September 12, 2026
This post builds intuition about the neural tangent kernel and shows how nonlinear models learn similarity.
Scaling Laws for a Linear Model
September 8, 2026
We derive analogues of compute-optimal scaling laws for a linear model.
What Kind of a Statistician is a Transformer?
August 19, 2026
A statistical view of in-context learning through Jacobians, smoothers, and effective degrees of freedom.
Transformer Induction Heads
July 30, 2026
Training small transformers to copy from context, and watching induction heads form as a two-layer circuit.
Much of linear least-squares learning is spectral filtering
July 23, 2026
OLS, ridge, PCR, gradient descent, momentum, and early stopping are all one estimator with a different filter function on the data spectrum.
Epoch-wise double descent in OLS
July 21, 2026
Double descent in plain OLS, traced over training epochs rather than model size, where the implicit bias of gradient descent does the work.