Subcategory
Random variables, Bayes' rule and maximum likelihood — the reasoning that turns noisy data into a loss function.
Nearly every model in the next two categories is fit by maximising a likelihood, whether or not its documentation says so. This category makes that explicit, starting from what a random variable is and ending at the mathematical statement of why more complex models trade bias for variance.
Read alongside Information Theory if you want the full derivation of where a loss function's specific shape comes from.
Bayes' theorem is a two-line consequence of how conditional probability is defined, but nearly all of its practical value comes from reversing the direction of conditioning — going from how likely evidence is given a cause to how likely a cause is given the evidence.
Behind the familiar "underfitting versus overfitting" picture is an exact algebraic decomposition of expected squared error into three terms — bias, variance and irreducible noise. This post derives that decomposition and shows precisely what choosing "more complex" is trading away, in exchange for what.
Maximum likelihood estimation is the method behind nearly every loss function in this corpus, whether or not a model's documentation says so. This post derives the method generally, then shows how assuming Gaussian or Bernoulli noise reproduces squared error and cross-entropy exactly.