Chapter 82·Beginner·11 min read
Training a Neural Network, End to End: From Random Dials to Reading Digits
Watch a neural network get trained from scratch, in plain English. We take a digit-reading network from random weights to 98% accuracy — forward pass, loss, backpropagation, batches and epochs — and see what training actually looks like in practice.
July 7, 2026
We've built the machine on paper: a stack of layers made of weighted votes with biases and bends, holding everything it knows in millions of adjustable dials. Time for the payoff. In this chapter we switch the machine on and train it — start to finish, on a real, classic task — and watch random noise turn into competence.
No math, as promised. But by the end, sentences like "we trained a model for 20 epochs and got 98% test accuracy" will read like plain English to you.
The task: read handwritten digits
Meet the fruit-fly of deep learning: MNIST, a famous dataset of 70,000 small grayscale photos of handwritten digits — 0 through 9 — each labeled with the right answer by a human. Sloppy 7s, cramped 4s, ambiguous 9s; scanned from real forms decades ago and used to teach every generation of ML students since.
Our network for the job, following the anatomy from the last two chapters:
- Input layer: each image is 28×28 pixels → 784 numbers (brightness of each pixel), because everything must become numbers first.
- Two hidden layers — 128 neurons, then 64 — the feature-builders.
- Output layer: 10 neurons, one per digit. Their outputs are turned into percentages (softmax, from last chapter's table) — the network's confidence in each digit.
All in, roughly 100,000 parameters — a toy by modern standards, plenty to read digits well.
First, we split the data and lock part of it away:
| Split | Size | Job |
|---|---|---|
| Training set | 60,000 images | The network learns from these |
| Test set | 10,000 images | Never shown during training — the final exam |
That held-out test set is sacred. As the ML guide covers in depth, a network judged on the data it trained on can cheat by memorizing; only fresh examples reveal real learning.
Step zero: random dials, dice-roll guesses
Every one of the 100,000 weights and biases is initialized to a small random number. The network at this moment knows nothing — it is static, noise, a shrug shaped like a function.
Feed it its first training image — say, a handwritten 7 — and the numbers flow through: pixels weighted and summed, biases added, bends applied, layer to layer, out to ten confidence scores. This left-to-right flow has a name you'll see everywhere: the forward pass. The fresh network's verdict on our 7:
About 10% on everything. Pure chance. And this is exactly the right starting point, because training doesn't need a good network — it needs a wrong one, plus a way to measure the wrongness.
The loop that does everything
Here is the entire algorithm of deep learning, the loop that takes this network from dice-rolls to 98%:
Guess. The forward pass, as above: image in, ten confidences out. For our 7 it said "3, probably."
Score the error. Compare the guess against the label. The right answer was 7; the network gave 7 only 9% confidence. All that disagreement is crushed into a single number — the loss — meaning simply how wrong was that. Terrible guess, big loss; confident correct guess, loss near zero. The loss is the only feedback the network will ever receive: not "that loop on top means it could be a 2," just a score. Wrongness, quantified.
Assign blame. The hard part. The error is one number at the output, but it was produced by 100,000 dials acting together — which ones should change, and which way? Backpropagation answers this: it runs backwards through the layers, splitting responsibility for the error among every weight and bias according to how much each contributed. The dial that pushed hard toward "3" gets strong blame; a barely-involved dial gets a whisper.
Nudge. Every dial shifts a tiny step in its blame-reducing direction. Not a rewrite — a nudge. The step size (the learning rate) is kept deliberately small: each individual example is messy and shouldn't get to yank the dials around; learning is the accumulation of millions of gentle corrections, not a few dramatic ones.
Repeat. Next examples, same loop, until the loss stops falling.
If you want the deeper intuition behind those last two beats — why nudging works at all, the beautiful "walking downhill in fog" picture of gradient descent, and how backprop's blame-splitting made deep networks trainable — we've dedicated a whole chapter to it in the ML guide: How neural networks learn. Here, we stay on the factory floor and keep training.
Batches and epochs: the bookkeeping words
Two terms of art appear in every training story, and both are simple:
- Batch. In practice the network doesn't nudge after every single image — it processes a small stack of examples (say 32 or 64) in one go, and makes one combined nudge from their average error. Smoother feedback, and vastly faster on GPUs, which love doing the same arithmetic on many things at once.
- Epoch. One full pass through the entire training set. With 60,000 images in batches of 64, an epoch is roughly 940 guess-score-blame-nudge cycles. Then the data is shuffled and the network goes through it all again.
So the sentence "we trained for 20 epochs with batch size 64" decodes to: the network saw every training image 20 times, updating its dials after every 64 images — about 19,000 nudges of all 100,000 dials. Modest numbers; frontier LLMs run the same loop with billions of dials over trillions of words on thousands of GPUs for months. Same five boxes. More electricity.
Watching it learn: the loss curve
What does a practitioner actually see during those 19,000 nudges? One chart, watched like a heart monitor: loss over time.
The shape tells the story. Early epochs are steep: from random dials, almost any nudge helps, and easy patterns get absorbed fast (a blank left column? probably a 1). Later epochs flatten out: what remains are the genuinely hard cases — the 4s that look like 9s — where progress is scraped out slowly. When the curve goes flat, training has found what it's going to find.
Run the exam — the 10,000 test images it has never seen — and our little network scores around 98%. From dice-roll to reading handwriting in twenty passes over the data.
What the 2% teaches: how training goes wrong
Our network misses about 200 of the 10,000 test digits, and the failure patterns are the same ones that haunt billion-dollar models:
- Genuinely ambiguous inputs. Some of those digits are scrawls humans argue about too. No amount of training fixes an unreadable 4-or-9; the data ceiling is real.
- Memorization (overfitting). Train too long or with too big a network, and training loss keeps falling while test accuracy quietly gets worse — the network is memorizing quirks of the 60,000 instead of learning digit-ness. This is why the test set exists, and why an entire chapter of the ML guide is devoted to it.
- Learning the wrong shortcut. The loop rewards whatever reduces loss. If some quirk of the scanning process correlated with certain digits, the network would happily learn the quirk instead of the shape. It optimizes the score, not your intent.
- Confident nonsense. Feed the trained network a doodle of a cat and it must still answer in digits — perhaps "8, 91%." It has no concept of "not a digit" unless we build one. Sound familiar? It's the small-scale ancestor of why LLMs hallucinate.
None of this is a bug in the loop. It's what "reduce the error on the training data" honestly delivers — no more, no less.
Recap
- Training data gets split: a big training set to learn from, a locked-away test set for the final exam.
- Networks start with random weights and dice-roll guesses; training needs only wrongness and a way to score it.
- The universal loop: forward pass → loss → backpropagation assigns blame → nudge every dial → repeat. (Deep-dive on why this works.)
- A batch is the handful of examples per nudge; an epoch is one full pass through the training data. "20 epochs, batch 64" is now plain English.
- Progress is watched on the loss curve — steep early learning, hard-won gains later, flat when done.
- The failure modes — ambiguity, memorization, shortcuts, confident nonsense — come from the same source as the successes: the loop optimizes the score, not understanding.
One thing we quietly glossed over: our network treated each image as 784 independent numbers — it never knew pixels had neighbors. For images, sound, and language, the shape of the data matters enormously, and that's why neural networks come in different architectures. Final chapter: the neural network family tree — CNNs, RNNs, and Transformers.