Subcategory
Optimisers, normalisation and initialisation — the choices that decide whether a network trains at all, not just what it computes.
Two networks with an identical architecture can behave completely differently depending on how they are trained, which is what makes training dynamics its own subject rather than a footnote to architecture.
This subcategory covers the parts of a training run that are invisible in a diagram: the optimiser doing the updating, the normalisation keeping activations in a workable range, the regularisation fighting memorisation, and the initial weights that decide whether any of it gets off the ground.
Batch normalisation re-centres and rescales a layer's activations using statistics from the current batch, and that simple operation removed much of the fragility that made deep networks hard to train. This post derives the operation, its effect on the loss landscape, and the difference between training-time and inference-time behaviour.
Dropout randomly disables units during training, forcing the network to avoid depending on any one of them too heavily; weight decay penalises large weights directly. This post derives both as regularisation strategies with different mechanisms but the same goal, and compares their effect on what the model actually learns.
Stochastic gradient descent's noisy, single-direction updates cause nameable problems — oscillation across narrow valleys, no adaptation to parameters needing different step sizes — and momentum and Adam are direct fixes to those problems, not arbitrary improvements. This post derives each in terms of the failure it corrects.
Initialised too small, activations shrink to nothing layer by layer; initialised too large, they explode — both failures happen before a single gradient update improves anything. This post derives the variance-preserving reasoning behind Xavier and He initialisation, and why the right scale depends on the activation function chosen.