Weights, Biases & Activation Functions: Inside a Single Neuron

In the last chapter we saw the big picture: a neural network is a giant adjustable function — layers of tiny voting units, with all its knowledge stored in dials called weights. Now let's zoom all the way in. What exactly is one of those units?

This matters because a neural network is one design repeated everywhere. There is no exotic machinery hiding in layer 47 — it's the same little unit, copied millions of times. Understand one neuron and you understand the substance of the whole network. And one of its three parts — the humble activation function — turns out to be the load-bearing trick of all deep learning.

The three moving parts

Every artificial neuron processes its inputs in the same four beats:

Multiply each input by its weight

↓→

Add them all up

↓→

Add the bias

↓→

Apply the activation function

Three kinds of parts appear in that recipe — weights, one bias, and one activation function. Let's take them one at a time, with a running example: a single neuron inside a spam filter, receiving signals like "contains the word FREE", "sender is in your contacts", and "has suspicious link."

Weights: the importance dials

Each input arrives through a connection, and each connection carries a weight — a number that scales the signal on its way in.

Weight value	Meaning for the neuron
Large positive (say +4)	"This input is strong evidence for firing"
Small positive (+0.3)	"Mild evidence for"
Near zero	"Irrelevant — ignore this input"
Negative (−5)	"This input argues against firing"

Our spam neuron might settle on: suspicious link → +4, word FREE → +2, sender in contacts → −5. Now a message with a shady link and "FREE!!!" from a stranger produces a strongly positive total, while the same message from your mom gets dragged negative by that −5. The neuron's entire "opinion" about spam is encoded in three numbers.

Scale that up and you have the network: the weights are where everything the model knows lives. When training nudges a network toward fewer mistakes, weights are what's being nudged. Nothing else in the neuron changes shape — the recipe is fixed; only the dial positions move.

Bias: the trigger threshold

The bias is one extra number added to the total before the neuron decides anything — and the cleanest way to read it is as the neuron's threshold setting:

• Big negative bias → a skeptical neuron. Inputs must supply a lot of positive evidence before it fires. ("Assume it's not spam unless the evidence is loud.")
• Positive bias → a trigger-happy neuron, halfway to firing before any evidence shows up.

Why does this need to be adjustable? Because without a bias, every neuron's dividing line would be forced to pass through the same neutral point — like judging every exam with the pass mark nailed at exactly 50%, forever. The bias lets each neuron choose its own pass mark. It's a small part, but every neuron carries one, and training tunes it right alongside the weights.

Activation functions: the bend that makes depth work

After weighing, summing, and adding the bias, the neuron holds one number. The last beat passes it through the activation function — a fixed little rule that reshapes the number before it's sent onward. The most famous ones:

Function	The rule, in words	Where you'll meet it
ReLU	Negative? Output zero. Positive? Pass unchanged.	The default workhorse in most modern networks
Sigmoid	Squash anything into 0–1	Outputs that mean probability ("87% spam")
Tanh	Squash into −1 to +1	Older networks, some sequence models
Softmax	Turn a list of scores into percentages summing to 100%	Final layer of classifiers — and how an LLM scores next tokens

ReLU deserves a pause, because its silliness is the point. The rule is literally "if negative, say zero; otherwise, repeat the number." That's the trick powering most of deep learning. Why on earth does anything so trivial matter?

Why the network needs the bend

Here's the punchline of this chapter. Weighing and adding are straight-line operations — feed them straight lines, and straight lines come out. And straight-line operations have a fatal property: stacking them achieves nothing. Chain any number of weigh-and-add layers together and the result is mathematically identical to one weigh-and-add layer. A 100-layer network would collapse, like a telescope, into a 1-layer network. Depth would be a lie.

The activation function is what prevents the collapse. By putting a bend — a kink, a squash, any nonlinearity — after every neuron, each layer becomes more than a re-mix of straight lines, and stacking layers becomes genuinely, irreversibly more powerful. This is precisely what unlocks the layer-by-layer story from chapter one: edges → textures → parts → "dog" is only possible because each floor of the building adds its own bends.

Think of drawing with straight rulers versus bendy ones. With straight rulers, no matter how many strokes you layer, you can only ever produce straight lines. Give every stroke a slight bend, and suddenly enough strokes can trace any curve — a face, a coastline, the boundary between spam and not-spam. ReLU is the cheapest possible bend, and cheap matters when you're applying it billions of times.

Watching one neuron think

Let's run our spam neuron end to end on a real message — "FREE crypto rewards!!" with a shady link, from an unknown sender:

1. Weigh: suspicious link (yes) × +4 = 4. Word FREE (yes) × +2 = 2. Sender known (no) × −5 = 0.
2. Add: total evidence = 6.
3. Bias: the neuron is skeptical, bias −3 → running total 3.
4. Activate: ReLU of 3 → 3. A confident positive signal heads to the next layer.

Now your mom emails "FREE cake at mine on Sunday": FREE contributes +2, sender in contacts contributes −5, bias −3 → total −6 → ReLU outputs 0. Silence. The neuron simply doesn't fire, and downstream layers never hear a whisper of spam-suspicion about cake.

That silence is worth noticing: a neuron's way of saying "not my pattern" is to output nothing at all. At any moment, a large share of the neurons in a deep network are silent — each one is a specialist that speaks up mainly when its particular pattern shows up. What downstream layers receive is, in effect, a chorus of just the specialists who recognized something.

One honest caveat: real neurons inside big networks don't get tidy labels like "suspicious link." Their inputs are outputs of other neurons — summaries of summaries — and what a given unit actually responds to can be strange, blended, and hard to name. The mechanics are exactly as above; the interpretations are usually murkier. That opacity is a running theme we'll return to at the end of the guide.

From one neuron to a billion parameters

Zoom back out with fresh eyes. A layer is hundreds of these weigh-add-shift-squash units running side by side on the same inputs, each with its own weights and bias — so each learns to respond to a different pattern. The next layer does the same over their outputs. Every arrow in the network diagram is a weight; every unit adds a bias and a bend.

Count it up for a modest image network: tens of millions of dials, each one individually tuned. Which resurfaces, more urgently than before, the question this guide keeps circling: who on earth sets all these dials? No human could. The answer — the automatic dial-tuning loop that is the actual "learning" in machine learning — is the next chapter, where we train a digit-reading network end to end and watch the dials find their own settings.

Recap

• Every neuron runs the same four beats: weigh each input, add them up, add the bias, apply the activation function. The whole network is this unit, repeated.
• Weights are importance dials: positive amplifies, negative argues against, near-zero ignores. They're where all learned knowledge lives.
• The bias is the neuron's own trigger threshold — how much evidence it demands before speaking up.
• Weights + biases = parameters. "70B parameters" = 70 billion of these dials.
• Activation functions add the essential bend: without them, stacked layers collapse into a single layer and depth is worthless. With them, enough simple bends can trace any boundary.
• ReLU — zero out negatives, pass positives — is the deliberately-cheap bend that powers most of modern deep learning.

Next: we stop admiring the machine and switch it on — training a neural network end to end, from random dials to reading handwritten digits.