Subcategory
Regression, classification and the ensembles built from both — every model here trained on labelled examples.
Linear regression and logistic regression between them introduce the feature vector, the weight vector, the loss, the gradient and the regulariser — the full vocabulary that every model in this subcategory reuses.
Decision trees and the forests built from them approach the same problem from a completely different direction: no gradient, no closed form, just recursive splitting — and support vector machines return to an optimisation view, from the geometry of the margin rather than the likelihood.
Unlike every model derived so far in this corpus, a decision tree isn't fit by an optimiser — it's grown by recursively splitting on whichever question reduces impurity the most. This post derives the impurity measures behind that choice and where a greedy tree's structure comes from.
Where logistic regression asks for a probability, a support vector machine asks a purely geometric question — what is the widest possible margin between two classes, and which points define it? This post derives the margin-maximisation objective, support vectors, and a first look at the kernel trick.
Random forests and gradient-boosted trees dominate classical machine learning competitions, and both are built from the same weak base learner combined two structurally different ways. This post derives bagging as variance reduction through averaging, and boosting as sequential error correction.