Subcategory
The neuron, the backward pass, and the non-linearity between them — the ideas every architecture later in this category assumes.
Before architectures, before frameworks, before any of the parts that change every eighteen months: a unit that computes, the non-linearity that makes stacking those units worthwhile, a method for assigning blame across a stack of them, and the history of how that stack was arrived at.
Worth working through by hand once. After that you can let a framework do it and still know what it is doing.
An artificial neuron is a weighted sum followed by a non-linearity, which makes it a logistic regression that has stopped claiming to model a probability. This post explains what the non-linearity is actually for, why stacking linear units gains nothing, and how a layer becomes a matrix multiply.
Every activation function solves the same structural problem — making depth meaningful — but they differ enormously in gradient behaviour, and that difference decided which ones survived. This post compares sigmoid, tanh and ReLU on exactly that basis, and explains ReLU's dominance via the vanishing gradient problem.
The single perceptron's inability to represent XOR very nearly ended neural network research, and the resolution — stacking perceptrons into layers — is the entire origin of "deep" in deep learning. This post tells that history because the technical point behind it still explains what depth buys you today.