Activation Functions
ReLU, sigmoid, tanh, GELU. Nonlinearity is what stops a stack of linear layers collapsing into one; the choice decides which gradients survive the trip back.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Why does a network need a nonlinearity between its layers at all, and how does the choice of activation decide whether the gradients reach the early layers?
A team's six-layer network on sensor sequences trains no better than a two-layer one. "We added depth and got nothing. The training loss barely moves in the first few layers' weights, and half the units in layer three output zero on everything."
Activations are a detail. Pick one, stack more layers to get more capacity, and let gradient descent sort out the weights. Depth is what makes deep learning work.
The all-sigmoid network trains its last two layers and leaves the first four nearly at initialisation. Each sigmoid's derivative is at most one quarter, and the product of six of them on the way back is tiny — the gradient that reaches layer one is a rounding error (Vanishing and Exploding Gradients).
- The all-sigmoid network trains its last two layers and leaves the first four nearly at initialisation. Each sigmoid's derivative is at most one quarter, and the product of six of them on the way back is tiny — the gradient that reaches layer one is a rounding error (Vanishing and Exploding Gradients).
- The ReLU network with the high learning rate kills a third of its units in the first few hundred steps: a large update drives their biases negative, their outputs go to zero on every input, and their gradient is zero thereafter. Capacity that was paid for is gone, silently.
- Neither failure shows in the loss curve as an error; both show as "the deeper network does no better", which the team read as "depth does not help on this data".
- The validation metric was computed on the trained network and looked no worse than the shallow one, so the pipeline promoted a six-layer model whose first four layers are noise, at three times the serving cost.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an industrial pump will fail within the next week from a sequence of vibration readings. The label is the maintenance event, arriving when the pump is actually serviced.
- The decision is whether to schedule a pre-emptive inspection; a missed failure is downtime, a false alarm is a wasted technician visit.
- One example is one pump-week: a few thousand sensor readings summarised into a few hundred features, and the failure-within-a-week label. Tens of thousands of pump-weeks.
- Features are standardised; the label is rare — a few percent of pump-weeks.
- The first network used sigmoid activations throughout because that is what the tutorial used; the second used ReLU with a default initialisation and a high learning rate.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A composition of linear maps is a linear map: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂). However deep the stack, without a nonlinearity between layers the network can only represent what a single linear layer can. The activation is what makes depth mean anything.
- Sigmoid, σ(z) = 1 / (1 + e^(−z)), maps to (0, 1) and saturates: its derivative σ(z)(1 − σ(z)) peaks at one quarter and vanishes for |z| large. tanh is a rescaled sigmoid to (−1, 1), zero-centred, derivative at most one. Both shrink every gradient that passes through them; stacked, they shrink it geometrically.
- ReLU, max(0, z), has derivative one for positive z and zero otherwise. Gradients pass through active units unchanged, which is why deep ReLU networks train at all. The cost is the dead unit: once z is negative on every input, the derivative is zero everywhere and the unit cannot recover. Leaky ReLU gives a small slope for negative z so the gradient is never exactly zero.
- GELU and similar smooth variants — z·Φ(z), where Φ is the Gaussian CDF — behave like ReLU for large |z| but curve smoothly through zero, so small negative pre-activations still pass a small gradient. Transformers use GELU almost universally; the practical differences from ReLU are small and empirical (Transformer Fundamentals).
Why the nonlinearity is not optional
The algebra is short enough to keep in your head. A layer without an activation is x ↦ Wx + b. Two of them composed give W₂(W₁x + b₁) + b₂, which is (W₂W₁)x + (W₂b₁ + b₂) — another matrix and another vector. Ten of them are still one matrix and one vector. The stack has more parameters than a single layer and exactly the same expressive power.
Put any nonlinear function between the layers and the composition is no longer linear: the second layer sees a bent version of the first layer's output and can carve the input space along curves. That bend is the whole mechanism by which a network represents interactions the inputs do not contain.
1import numpy as np2rng = np.random.default_rng(0)3x = rng.normal(size=8)4W1, b1 = rng.normal(size=(16, 8)), rng.normal(size=16)5W2, b2 = rng.normal(size=(4, 16)), rng.normal(size=4)6 7stacked = W2 @ (W1 @ x + b1) + b28collapsed = (W2 @ W1) @ x + (W2 @ b1 + b2)9assert np.allclose(stacked, collapsed) # depth bought nothing10 11relu = lambda z: np.maximum(0, z)12bent = W2 @ relu(W1 @ x + b1) + b2 # no single (W, b) reproduces this for all xThe assertion is the lesson: the sixteen-unit hidden layer disappears. The one line with relu is the only place the network stops being a logistic regression with extra steps.
Which gradients survive the trip back
Backpropagation multiplies local derivatives along the path from the loss to each parameter. Every activation on the path contributes its derivative as a factor. A sigmoid contributes at most one quarter, a tanh at most one, a ReLU exactly one when active and exactly zero when not. Six sigmoids in a row contribute at most (1/4)⁶ — and typically far less, because most units are not at the peak of their derivative.
So the activation choice is a choice about which parameters can be reached by the training signal. ReLU's derivative of one is why deep networks became trainable; its derivative of zero is why they can lose units for good. The smooth variants trade a tiny compute cost for a derivative that is never exactly zero.
| Activation | Range | Derivative | Failure | Typical place |
|---|---|---|---|---|
| sigmoid | (0, 1) | σ(z)(1 − σ(z)), at most 1/4 | Saturates; gradients vanish when stacked | Binary output only |
| tanh | (−1, 1) | 1 − tanh²(z), at most 1 | Saturates; zero-centred helps a little | Gates in recurrent cells; legacy hidden layers |
| ReLU | [0, ∞) | 1 if z > 0 else 0 | Dead units: zero gradient forever | Default hidden activation for dense and conv stacks |
| leaky ReLU | (−∞, ∞) | 1 if z > 0 else small α | Rarely dies; no exact zeros for pruning | Where dead units were observed |
| GELU | ≈ (−0.17, ∞) | smooth, ReLU-like for large |z| | Slightly more compute; behaviour empirical | Transformer feed-forward blocks |
What to record, and what must hold
Both failures — vanished gradients and dead units — are invisible in the loss curve and visible in two cheap statistics: the gradient norm per layer, and the fraction of active units per layer. A run that logs them cannot promote a six-layer network with four decorative layers.
After deployment the activation regime is an assumption like any other. A ReLU layer that was active on a third of inputs is expected to stay near that; an output sigmoid that operated in its steep region is expected to stay there. Input drift moves both, and both are measurable without labels.
At serving time the fraction of active ReLUs per layer and the magnitude of the output pre-activation are close to their end-of-training values.
holds when Inputs are standardised by the training statistics, normalisation layers use the correct serving statistics, and the input distribution has not drifted far.
breaks when Input drift pushes pre-activations into saturation or below zero for whole layers; a normalisation layer is served with training-mode batch statistics; a retrain with a different learning rate kills units that were carrying the decision.
respond Treat a regime shift as input drift first — check scaling and normalisation statistics — and as a retrain problem second; never as a reason to add depth.
How to build it
Most important first.
- Use ReLU-family activations in hidden layers by default; reserve sigmoid for an output that must be a probability and softmax for an output over classes. The activation at the output is dictated by the loss, not by taste (Loss Functions).
- Initialise weights for the activation in use — He initialisation for ReLU, Xavier for tanh — so that pre-activations start in the range where the derivative is meaningful (Initialisation and Convergence).
- Pair depth with normalisation layers and, past a handful of layers, residual connections, which give the gradient a path back that does not pass through every activation (Normalisation Layers).
- Record per-layer activation statistics — fraction of active ReLUs, mean |z| for saturating units — as first-class training diagnostics next to the loss.
- Tune the learning rate with dead units in view: the highest rate that trains fastest is often one that kills units early (Batch Size and Learning Rate).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-layer gradient norm during training. If layer one's gradient norm is orders of magnitude below layer six's, the early layers are not learning and depth is decorative.
- Fraction of dead ReLUs per layer at the end of training, over a validation batch — capacity lost.
- Validation recall at the inspection-capacity threshold for the deep and shallow networks on a time-based split; if depth does not move it, do not pay for depth (Time-Based Split).
- Do not measure "the deep network reached a lower training loss" as evidence that the depth helped; memorisation in the last two layers can do that alone.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Pre-activations at serving time stay in the range where the activation was operating during training — a ReLU that was active on a fraction of inputs stays active on a similar fraction.
- The units alive at the end of training are the ones the prediction depends on, and no later fine-tune or retrain has killed them.
- Normalisation layers use their serving-time statistics correctly, so the activations see the distribution they were trained on.
- Inputs remain standardised so a saturating output is not pushed into its flat region by scale alone.
- Offline: per-layer gradient norms and activation rates logged during the run; a deep network whose early-layer norms are negligible fails the run's smoke test (Training Smoke Tests).
- Online: a serving-side check that the hidden activations for a canary input match the training-time values, which catches wrong normalisation statistics (Serving Contract Tests).
- Over time: the serving distribution of output pre-activations and the fraction of active units on sampled traffic; a drift towards saturation or towards all-dead is an input-distribution change.
What can go wrong
- Normalisation layers fix the training-time gradient flow and introduce a train/serve difference: batch statistics at training, running statistics at serving, and a serving bug that uses the wrong ones.
- Leaky ReLU keeps units alive and makes the network slightly harder to prune later, because no unit is exactly zero.
- A learning-rate schedule fixed the dead-unit problem for one dataset size; the next retrain on three times the data with the same schedule kills units again.
- A sigmoid output on rare labels saturates at zero for the common class and the loss stops producing gradient for the rare one; the model underpredicts failures and the loss looks fine (Class Imbalance).
- ReLU gives gradients a clean path and dead units a permanent grave; the smooth variants avoid the grave at a small compute cost and with slightly different, empirically tuned behaviour.
- Normalisation and residual connections let depth train and add components whose serving-time behaviour must be tested separately from the weights.
- Recording activation statistics is cheap and rarely done; it converts two silent failures into visible ones.
- "The activation is just a detail; capacity comes from depth." Without the activation, depth is a factored linear model. With a saturating activation, depth is where the gradient goes to die.
- "Half the units are dead, but validation is fine, so it does not matter." It means you are serving a network at the cost of N units and using half of them. A smaller network would be cheaper and no worse.
- "Use sigmoid everywhere because outputs should be probabilities." Only the output layer needs to be a probability. Sigmoids in hidden layers are why deep networks did not train for years.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe collapse of stacked linear layers into one is algebra and holds for every architecture; the choice of activation and its effect on gradient flow applies wherever layers are composed, dense, convolutional or attention.
- MODEL-SPECIFICWhich activation to default to depends on the architecture: ReLU or its leaky variants for dense and convolutional stacks, GELU inside transformer blocks, sigmoid or tanh where an output or a gate must be bounded; the reasons are empirical and the differences among ReLU-family choices are usually small.
- SIMPLIFIEDFractions such as "a third of units dead" and "derivative at most one quarter" are for the shape of the argument; the first is illustrative, the second is exact for the sigmoid but the compounding across layers depends on the weights as much as on the activation.
Where the depth lives
This domain teaches the model and hands the rest off by name.