Small model choices still matter

def relu(x):
    return max(0, x)

That function is small enough to look harmless. It is also one of the places where a neural network decides which signals are allowed to continue.

After a layer calculates a weighted sum, the activation function bends the result before the next layer sees it. Without that bend, stacking linear layers still gives you one larger linear transformation. The network can multiply matrices all day and never learn the kind of changing boundary that makes a hidden layer useful.

ReLU makes the choice visible. Negative values become zero. Positive values pass through. The result is often easier to optimize than a saturating function, and many activations become sparse. The cost is equally concrete: units that spend too much time below zero can stop contributing useful gradients.

I like teaching this with a few numbers instead of a diagram:

input:   [-2.0, -0.1, 0.3, 1.7]
relu:    [ 0.0,  0.0, 0.3, 1.7]

Two signals disappeared. That may be exactly what the task wants. It may also be the reason a small classifier misses a weak but important case.

The production mistake is to blame the dramatic part of the system first. A model misses an edge case, and the investigation jumps to the dataset, the prompt, or the serving stack. Those may be the right answers. So might the activation, normalization, threshold, or sampling default that quietly shaped the output before anyone saw it.

For a small model, I would run a plain ablation before reaching for a bigger one. Change the activation. Move the threshold. Remove one feature. Compare the false positives and false negatives, not only the average score.

Small models are not casual models. They just make the consequences of their small decisions easier to see.