Series

The Loss Function Toolkit

Three posts that derive the same loss functions from two independent directions — likelihood and information — and show why they agree.

Every loss function in this corpus is derived, never assumed — and this series is where that habit gets its full justification, from two directions that turn out to be the same direction.

The two routes

Statistical — assume a distribution generated the data, then maximise how likely the data is under it. That is maximum likelihood estimation, and it produces squared error and cross-entropy as special cases rather than as defaults.

Informational — measure the cost, in bits, of describing data with the wrong distribution instead of the right one. That is cross-entropy read through entropy and KL divergence, and it produces the identical loss functions from an entirely different starting point.

Why both, not just one

Seeing a loss function derived twice, from unrelated premises, is what makes "cross-entropy is the natural choice for classification" a fact rather than a convention worth just memorising.

The arc

Part 1 of 3Maximum Likelihood Estimation: Where Most Loss Functions Actually Come From

Maximum likelihood estimation is the method behind nearly every loss function in this corpus, whether or not a model's documentation says so. This post derives the method generally, then shows how assuming Gaussian or Bernoulli noise reproduces squared error and cross-entropy exactly.

Part 2 of 3Entropy, Cross-Entropy, and KL Divergence: The Information-Theoretic View of Loss

This post builds the information-theoretic account of loss from the ground up — entropy as the average surprise of a distribution, cross-entropy as that same quantity measured against a different assumed distribution, and KL divergence as the information lost when the assumed distribution is wrong.

Part 3 of 3Why Cross-Entropy Loss Is the Natural Choice for Classification

Two independent routes — maximising likelihood under a Bernoulli or categorical assumption, and minimising the information-theoretic cost of a wrong model — arrive at the identical loss function. This post lays both derivations side by side and explains why that convergence is not a coincidence.