Subcategory
Precision, recall, cross-validation and calibration — the discipline of knowing whether a model is actually good.
A model that fits is not the same as a model that works, and this subcategory is entirely about the gap between the two. It assumes the models from Supervised and Unsupervised Learning and asks the question neither of those subcategories can answer on its own: is this any good, and how do you know without fooling yourself?
Accuracy says nothing about whether a model's predicted probabilities mean anything. Calibration is the property that they do — that among predictions of 80% confidence, roughly 80% are correct — and this post derives how to measure miscalibration and the standard techniques for correcting it.
K-fold cross-validation is simple to implement and surprisingly easy to implement wrong — normalising before splitting, tuning hyperparameters on the folds used to report performance, or splitting time-ordered data randomly. This post derives correct cross-validation and walks through each of these leaks.
A classifier that predicts the majority class every time can have high accuracy and be useless. Precision and recall separate the two ways a model can be wrong, and this post derives both from the confusion matrix, along with the tradeoff between them that every threshold choice makes explicit.