A single neuron is less mysterious when you run the numbers

The useful thing about a single neuron is that nothing has time to become mysterious.

There are inputs, weights, and a bias. The neuron multiplies each input by its weight, adds the products and the bias, then passes the raw score through an activation function. The whole calculation is small enough to check by hand.

start with the raw score

Treat the neuron as one weighted question about the input.

The example input is:

x = [2, 1, 3]

The neuron uses:

weights = [1, -1, 1]
bias = -5

The raw score is:

z = 2 * 1 + 1 * -1 + 3 * 1 + -5

which gives:

z = -1

Nothing hidden happened. Each input was scaled by a weight, and the bias shifted the total. Z is the raw answer before an activation function changes it.

The signs tell you which direction each input pulls. A positive weight pushes the score up as its input grows. A negative weight pulls it down. A zero weight has no effect for this neuron.

The sign is easier to reason about than the size at first. Increase the first input and Z rises because its weight is positive. Increase the second input and Z falls because its weight is negative. Change a weight attached to an input of zero and this particular calculation does not move.

That last case is worth testing in the module. A parameter can matter to the model while having no effect on one selected input.

bias moves the baseline

Bias is not a spare term. It shifts the weighted sum before the activation function sees it.

Here the weighted input sum is:

2 * 1 + 1 * -1 + 3 * 1 = 4

With a bias of -5, the raw score becomes -1.

Change the bias to -3 and the same inputs and weights produce:

z = 1

That one change crosses zero and flips the ReLU output from inactive to active. The bias belongs to the neuron. It is part of the rule, not part of the input.

relu clips the negative side

ReLU is short for rectified linear unit. The name is less useful than the rule:

The rule is simple:

if z > 0, output z
if z <= 0, output 0

In the module, A is the value after ReLU. If Z is -1, A becomes 0. If Z is 3, A stays 3.

The rule is small, but it gives the neuron a non-linear step. Without activations, stacked layers would collapse into another linear transformation. For this example, the practical interpretation is enough: the neuron can stay silent for some inputs and pass a positive value for others.

The model is not impressive. It is inspectable. Its structure is:

• inputs feed the neuron
• weights scale the inputs
• bias shifts the sum
• Z shows the raw score
• ReLU turns Z into A

Four neurons are four versions of the same calculation. A hidden layer is a group of them. Matrix multiplication is the compact form. Backpropagation is what later adjusts the weights when the result is wrong.

Start the module by changing the bias until Z crosses zero. Then change one weight and predict the direction of movement before running the calculation. Change one input and identify the weight that controls its effect.

The useful result is not a dramatic output. It is being able to look at the numbers and know what should happen next.