Category
Networks of simple units, and the algorithm that makes training them possible at all.
Deep learning is often introduced as a break from what came before. It is not. A neuron is a model from the previous category with its statistical interpretation removed, and a network is a stack of them with a training algorithm good enough to make the stack worth having.
Starting from that continuity makes the rest of the field much less mysterious.
The neuron, the backward pass, and the non-linearity between them — the ideas every architecture later in this category assumes.
An artificial neuron is a weighted sum followed by a non-linearity, which makes it a logistic regression that has stopped claiming to model a probability. This post explains what the non-linearity is actually for, why stacking linear units gains nothing, and how a layer becomes a matrix multiply.
Every activation function solves the same structural problem — making depth meaningful — but they differ enormously in gradient behaviour, and that difference decided which ones survived. This post compares sigmoid, tanh and ReLU on exactly that basis, and explains ReLU's dominance via the vanishing gradient problem.
The single perceptron's inability to represent XOR very nearly ended neural network research, and the resolution — stacking perceptrons into layers — is the entire origin of "deep" in deep learning. This post tells that history because the technical point behind it still explains what depth buys you today.
CNNs, recurrent networks and the transformer — the structural choices that decide what a network can efficiently learn.
Attention replaces the assumption that information must pass step by step through a sequence with a direct, learned weighting over every position at once. This post derives queries, keys and values from the problem attention solves, rather than starting from the equation, and shows why removing recurrence removed the vanishing gradient problem with it.
Convolution is weight sharing applied to spatial data — the same small filter, reused at every position, on the assumption that a pattern useful in one part of an image is useful anywhere in it. This post derives the convolution operation, pooling, and why this bias reduces parameters without reducing what the network can represent.
A recurrent network applies the same weights at every step of a sequence, which lets it handle arbitrary length — and also causes its gradients to shrink or explode geometrically over long sequences. This post derives both the architecture and the vanishing gradient problem from the same repeated-multiplication structure.
Multi-head attention, positional encoding, residual connections and layer normalisation each solve a distinct problem, and the transformer is their combination rather than one idea. This post builds the architecture block by block, motivating each component before assembling the full encoder-decoder structure.
Optimisers, normalisation and initialisation — the choices that decide whether a network trains at all, not just what it computes.
Batch normalisation re-centres and rescales a layer's activations using statistics from the current batch, and that simple operation removed much of the fragility that made deep networks hard to train. This post derives the operation, its effect on the loss landscape, and the difference between training-time and inference-time behaviour.
Dropout randomly disables units during training, forcing the network to avoid depending on any one of them too heavily; weight decay penalises large weights directly. This post derives both as regularisation strategies with different mechanisms but the same goal, and compares their effect on what the model actually learns.
Stochastic gradient descent's noisy, single-direction updates cause nameable problems — oscillation across narrow valleys, no adaptation to parameters needing different step sizes — and momentum and Adam are direct fixes to those problems, not arbitrary improvements. This post derives each in terms of the failure it corrects.
Initialised too small, activations shrink to nothing layer by layer; initialised too large, they explode — both failures happen before a single gradient update improves anything. This post derives the variance-preserving reasoning behind Xavier and He initialisation, and why the right scale depends on the activation function chosen.
Autoencoders, GANs and diffusion models — three different answers to how a network learns to generate rather than only classify.
An autoencoder has no labels, no external objective beyond reconstructing what it was given — and the bottleneck it is forced through is what makes that reconstruction task a genuine learning signal. This post derives the architecture, the compression it performs, and how it generalises into the variational autoencoder.
Diffusion models start from a deliberately simple forward process — gradually adding noise until data becomes indistinguishable from static — and train a network to reverse it one small step at a time. This post derives the forward process, the reverse process it implies, and why generation by many small steps proved more stable than in one large one.
A generative adversarial network pits a generator, trying to produce convincing fake data, against a discriminator, trying to catch it, and trains both simultaneously as a minimax game. This post derives that game formally, the equilibrium it aims for, and the instability that made GANs notoriously difficult to train.