What learning is
A straight line, a bowl, a walk downhill, and a fresh week of data. Everything else in the series is these four ideas at scale.
- 01
What is a model?
A model is a rule with dials, and data sets the dials.
Play with tilt w, height b
Twelve days at a café · Draw a rule · How wrong is it? · Square the gaps, add them up · Now make the guess · Try to beat it · This is what a model is
Open explainer → - 02
The loss landscape
Every setting of the dials has an error; drawn over all settings it is a bowl.
Play with the loss shape δ
Every line has a score · Turn one dial · Turn both · It is a bowl · Change how you score, change the bowl · Walk the map · Name it: the loss landscape
Open explainer → - 03
Gradient descent
Feel the slope, step against it, repeat: you find the bottom without a map.
Play with learning rate η, start point
Standing in fog · Feel the slope · Take a step, then another · Let it roll · How far do you dare to step? · Why it zigzags · Name it: gradient descent
Open explainer → - 04
Train vs test
Zero training error means nothing; only data the model never saw can judge it.
Play with bendiness, days recorded
A better rule? · Bend the rule · A fresh week arrives · Two scores, one sweep · Play with it · Name it: overfitting
Open explainer →
The neuron and the network
From one straight cut to a network that winds itself around a spiral, with the blame-passing that makes it trainable.
- 05
One neuron
A neuron is three numbers making one straight cut across the plane.
Play with w₁, w₂, b
Two numbers in, one answer out · A neuron · The neuron is a line · Weights tilt, bias slides · Score it, then fix it · Play with it · Where one cut fails
Open explainer → - 06
Why nonlinearity
Stack straight cuts and you still get one; add a kink and they fold into shapes.
Play with mixing weights on three cuts
The ring, again · Stack them · Three cuts mixed is… one cut · Add a bend · Now the mix can fold · Four bends make a ring · Name it: the activation function
Open explainer → - 07
The hidden layer folds space
A hidden layer builds new coordinates in which the hard problem is a straight cut.
Play with fold positions, hidden units
Busy in the middle · Two hidden neurons · The plane folds · Now one straight cut works · More neurons, more folds · Move the folds · Name it: representation
Open explainer → - 08
Backpropagation
Blame flows backwards, multiplying at every edge: the chain rule as a picture.
Play with w₁, w₂, η
One example, one mistake · Who is to blame? · Pass the blame backwards · The chain rule · Take the step · Play with it · Name it: backpropagation
Open explainer → - 09
Training in motion
A real training run, scrubbable: a network learns two spirals in 800 steps.
Play with learning rate, width
Two spirals · Step zero · Watch it learn · Turn the learning rate · Turn the width · The held-out points · Name it: a training run
Open explainer →
Seeing
How nine shared numbers find edges, how edges become corners become shapes, and how words become points.
- 10
What is a convolution
Slide nine numbers across a picture; the output is a map of where a pattern occurs.
Play with kernel choice
A picture is numbers · A 3×3 window · Slide it · What the new picture shows · Pick a kernel · Nine dials, not a hundred · Name it: convolution
Open explainer → - 11
Feature hierarchy
Edges, then corners, then shapes, each layer seeing a wider patch than the last.
Play with layer, square size
A picture with things in it · Layer 1: edges · Layer 2: corners · Layer 3: a square · How much each layer sees · Change the square · Name it: a feature hierarchy
Open explainer → - 12
Embeddings as geometry
Words as points, relationships as directions; king − man + woman lands by queen.
Play with which two dials to view
A network cannot read · Give each word a few dials · Words become points · Meaning is a direction · Arithmetic on meaning · Look along other dials · Name it: embeddings, and where they come from
Open explainer →
Sequences and the transformer
A language model is one probability chart, picked from over and over; here is the machine that produces it.
- 13
Predict the next token
A language model is a next-token distribution; temperature reshapes it before the pick.
Play with temperature, dice seed
The whole game in one chart · Where the numbers come from · Pick one, then do it again · Temperature · Write at different temperatures · Tokens, not words · Name it: a language model
Open explainer → - 14
Attention, visualised
Query dot key, softmax, weighted values: how one token reads the others.
Play with which token is looking
One word looks at the others · A key and a query for every token · Score every pair · Softmax: scores into shares · Values: what actually gets mixed · Pick a token · Name it: an attention head
Open explainer → - 15
Why positional encoding
Attention is order-blind; a stack of sine waves gives every slot a fingerprint.
Play with position, anchor
Two sentences, one bag · Stamp each slot with a number? · Read position off a set of waves · The fingerprint · Add it to the word · Why waves and not something else · Name it: positional encoding
Open explainer → - 16
The transformer block
One residual stream, nudged by attention then a feed-forward, stacked N times.
Play with number of blocks
The stream · Sub-block one: attention nudges the stream · Sub-block two: the feed-forward · Add, then normalise · Stack the block · Out the other end · Name it: the transformer
Open explainer →
Beyond fitting
Why models vary, how an agent learns with no dataset, and what every probability in the series actually meant.
- 17
Bias and variance
Reroll the data and draw every fit: the spread is variance, the gap is bias.
Play with bendiness, points per dataset
One dataset is one roll of the dice · Fit, reroll, fit again · Twenty rerolls, one model · Now a bendy model · Slide between them · The trade-off, drawn · More data shrinks the fan
Open explainer → - 18
Reinforcement learning in a grid
No answers, only rewards; value trickles backwards through a table as the agent explores.
Play with exploration ε, discount γ
No dataset, only a world · Four numbers per square · Reward trickles backwards · Watch it learn · Explore or exploit · How far ahead it cares · Name it: reinforcement learning
Open explainer → - 19
Bayes and uncertainty
Bayes is fading out the days that do not match; every model output is a belief.
Play with prior and clue strengths
Before you look · A clue arrives · Keep only the days that match · The same thing as areas · Clue after clue · When the thing is rare · Name it: every chart was a belief
Open explainer →