ClassicalMODEL-SPECIFICSCALE-SPECIFICDATA-SPECIFIC

Support Vector Machines

The widest street between the classes, defined by the few points closest to it. Kernels make a linear boundary curve; scaling is mandatory; the cost grows badly with the number of rows.

Target & data What to measure What must stay true

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

The model is defined by a handful of boundary points and ignores everything else — when is that a strength, when does it make training unaffordable, and what does a kernel actually buy?

The problem

A materials lab classifies sensor traces from a production line as defective or acceptable. "We have about eight thousand labelled traces, forty summary features each, and the boundary is not a straight line. We need something that generalises from a small set and that a QA engineer can be told about in one sentence."

The obvious approach

Fit a linear classifier. When it underfits, switch to an SVM with an RBF kernel on the raw features, tune C and γ on a grid, and pick the best validation F1. The kernel makes the boundary curve, the margin makes it generalise, and eight thousand rows train in seconds.

Why it breaks

On unscaled features the RBF kernel's distance is dominated by the pressure integrals, which range in the thousands. The boundary the SVM finds is a boundary in pressure only; every other feature is noise at the kernel's length scale. Validation looked acceptable because pressure alone carries some signal.

How it breaks — usually after the offline metric looked fine

On unscaled features the RBF kernel's distance is dominated by the pressure integrals, which range in the thousands. The boundary the SVM finds is a boundary in pressure only; every other feature is noise at the kernel's length scale. Validation looked acceptable because pressure alone carries some signal.
The tuned γ is a statement about distance in the scaled space. When the second line comes on with a different sensor layout, the feature distributions shift and the same γ describes a different neighbourhood size — the boundary is now too wiggly or too smooth for the new data, with no retraining and no error.
The decision function is a distance from the boundary, not a probability. The team thresholded it at zero, which is where the margin is centred, not where the cost of a missed defect balances a wasted inspection (Thresholding).
The plan to retrain on the combined history of both lines — a few hundred thousand traces — stalls: kernel SVM training grows between quadratically and cubically with rows, and the job that took seconds now takes days.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Predict whether a manufactured part will fail final inspection from forty features extracted from its sensor trace. The label is the inspection result, available at the end of the line, an hour after the trace.
The decision is whether to pull the part for manual inspection. Pulling a good part costs a few minutes; passing a bad part costs a field return.

Data

One example is one part: forty numeric features — peak temperatures, pressure integrals, timing offsets — on very different scales, plus the pass/fail label. Eight thousand rows, roughly one in nine defective.
Traces were collected over four months from one line; a second line with a slightly different sensor layout is scheduled to use the same model.
Labels are clean for failures caught at inspection and unknown for parts that pass inspection but fail in the field.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

A linear SVM chooses the separating hyperplane that maximises the margin — the distance to the nearest training points of either class. Only those nearest points, the support vectors, determine the boundary; moving any other point does nothing. That is the source of both its robustness and its blind spots.
C controls how much margin violation is tolerated: large C insists on classifying training points correctly and gives a narrow, wiggly margin; small C accepts errors for a wider, smoother one. It is a regularisation knob with the same bias/variance reading as any other (Regularisation).
A kernel replaces the dot product between two feature vectors with a similarity function, which is equivalent to fitting a linear boundary in a higher-dimensional space without constructing it. The RBF kernel's similarity decays with squared distance at a rate set by γ, so γ is a neighbourhood size — which is why scaling is not optional.
Training solves a quadratic programme over pairs of training points, so memory and time scale with n² at best; prediction costs one kernel evaluation per support vector. For tens of thousands of rows it is fine; for millions, linear SVMs via stochastic solvers are the only affordable variant and the kernel is gone.

The margin, and who defines it

Most classifiers use every training point to fit the boundary. An SVM uses only the points that would move it — the support vectors sitting on or inside the margin. Points far from the boundary contribute nothing, which is why the model is stable against them and why a single mislabelled point near the boundary can rotate it.

The margin is a regularisation device: of all the boundaries that separate the training data, the widest street is the one least likely to be wrong about a slightly different sample. C sets how much the model will narrow the street to avoid misclassifying a training point.

UserLLMAgentToolDataDecisionHumanGuardrail

The hinge loss the margin comes from

1import numpy as np
2 
3def hinge_objective(w, b, X, y, C):
4    # y in {-1, +1}. A point contributes zero loss once it is beyond the margin,
5    # which is why only the points near the boundary (support vectors) shape w.
6    margins = y * (X @ w + b)
7    violations = np.maximum(0.0, 1.0 - margins)
8    return 0.5 * w @ w + C * violations.sum()
9 
10# large C: pay dearly for every violation -> narrow margin, fits training points
11# small C: accept violations             -> wide margin, smoother boundary

The first term is the width of the street (small ‖w‖ means a wide margin); the second is the price of the points inside it. Everything about C follows from that sum.

Kernels: a curved boundary without building the features

The dual form of the SVM only ever uses dot products between pairs of training points. Replace the dot product with any function that behaves like a similarity — the kernel — and the algorithm fits a linear boundary in whatever feature space that kernel implicitly defines. The RBF kernel corresponds to an infinite-dimensional space; the model never constructs it, it only evaluates similarities.

The consequence for engineering is that γ is a length scale in the *scaled* feature space. Set γ large and each support vector influences only its immediate neighbourhood — a spiky boundary that memorises. Set it small and the boundary flattens towards linear. Change the scaling of one feature, and the same γ describes a different neighbourhood.

Forty scaled features, eight thousand rows, nonlinear boundary

Option	Quality	Latency	Cost	Interpretability	Data needed	Operational	Note
Linear SVM / logistic							Underfits a genuinely curved boundary; the baseline the kernel must beat.
RBF SVM							Strong at this size; coupling of γ to scale and the n² training cost are its limits.
Gradient-boosted trees							Scale-invariant and grows with the data; needs more rows to match the SVM here.
Small neural network							Nothing here needs a learned representation; more tuning surface for no gain at eight thousand rows.

caveat The scores assume scaled features and honest validation; on unscaled features the RBF row collapses and the tree row is unchanged. "Quality" is a rank on this size of clean numeric data — at a few hundred thousand rows the boosting row moves up and the SVM row becomes untrainable within a nightly window.

Scaling, the second line, and what to watch

The model was tuned on one line's distances. The second line produces features with different means and spreads, so under the same scaler its parts are systematically displaced in kernel space, and the boundary tuned for line one is applied to a cloud of points that has moved. No metric moves offline, because offline is line one.

What must stay true is that the scaled features at prediction time sit where the training fold's did. That is checkable per feature, without labels, and it is the monitor a kernel model needs more than a tree does.

must stay trueThe kernel's distances still mean what they meant

Every feature at prediction time, after the training-fold scaler, has a distribution close to what the support vectors were selected under, so γ describes the same neighbourhood.

holds when Sensors are unchanged and calibrated, the same scaler is applied, and any new line has been validated in shadow with its own decision-value distribution compared to the first.

breaks when A sensor is recalibrated or replaced, a new line with a different layout is routed through the same model, or the scaler is refitted on pooled data that no single line produces.

how you would know Per-feature drift of scaled mean and spread against the training fold; the decision-value distribution per line; the fraction of parts whose nearest support vector is farther than any training point's was.

respond Do not lower the threshold to keep the pull rate up. Re-validate scaling per line, retrain with the new line's data in the fold, and re-tune γ — the neighbourhood size has changed.

How to build it

Most important first.

Standardise every feature on the training fold and ship the scaler in the artifact; with an RBF kernel, the scaling decides what the model can see (Preprocessing Lives in the Artifact).
Tune C and γ jointly on a validation split that respects the data's structure — by time, or by line — and report the operating-point metric, not the default threshold (Grid Search and Random Search, Threshold Selection).
Choose the threshold on the decision function from costs; if a probability is needed, calibrate the decision values on held-out data rather than reading them as one (Calibration).
Start linear. A linear SVM or logistic regression on scaled features is the baseline the kernel must beat, and on forty features it often does not lose by much (The Linear Baseline).
Plan the model's ceiling: if the training set is going to grow past a few hundred thousand rows, the kernel SVM will not follow it, and a tree ensemble on the same features is the likely successor (Gradient Boosting).

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

Recall of defects at the inspection-capacity threshold — how many of the parts the line can afford to pull actually needed pulling. This is the number that maps to field returns.
Validation performance as a function of γ, on scaled features, to see whether the boundary is in a stable regime or on a knife-edge where a small shift in the features changes it.
Number of support vectors as a fraction of training points. Close to all of them means the model has memorised the set and the margin is not doing anything.
Do not read the decision-function value as a probability of defect; it is a scaled distance from the boundary whose calibration is accidental.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

Features at prediction time are scaled by the statistics of the training fold, and no feature has shifted by enough to change the distances the kernel computes.
The support vectors — the boundary parts — are still representative of where defective and acceptable parts meet; a process change that moves the boundary invalidates the model even if the bulk of the data is unchanged.
The defect base rate is close to the training rate, or the threshold has been re-set for the new rate.
The training set stays within the size the kernel solver can retrain in the pipeline's time budget.

How to verify — offline, online, and over time

Offline: nested validation over C and γ with scaling inside the fold; compare against a scaled linear model and a small tree ensemble at the same operating recall.
Online: shadow the second line for a week before routing its parts on the model's decisions, and compare the decision-value distribution to the first line's (Shadow Deployment).
Over time: monitor per-feature mean and spread against the scaler's statistics; alert when any feature drifts by more than a fraction of its training standard deviation, because in kernel space that is a moved boundary (Feature Drift).

What can go wrong

Failure modes in production

The scaler was fitted on both lines' data pooled, so the second line's features are scaled by statistics that include the first, and the model sees a distribution neither line produces.
Sensor recalibration shifts one feature by a constant; the kernel, working in distances, sees every part as far from every training point, and the decision function flattens towards the bias term.
The retraining job on the larger set times out and the pipeline silently keeps the old model, trained on one line, in production for both.
The threshold was tuned for one-in-nine defect rate; a process change halves the rate and the same threshold now pulls far more good parts than inspection can absorb.

What the recommended approach costs

Maximum margin gives good generalisation from a few thousand rows with no feature engineering beyond scaling; the price is a model whose training cost forbids growing with the data.
Kernels let a linear method fit a curved boundary without designing features; the price is a second hyperparameter that couples to the scaling and cannot be understood without it.
The one-sentence explanation — "the parts closest to the boundary define it" — is honest, but it does not tell the QA engineer which feature pushed a specific part over, the way a tree or a linear model would.

Misreads

"The kernel found nonlinear structure the linear model missed." Or the kernel found a boundary in the one unscaled feature that dominates the distance. Scale first, then compare.
"SVMs generalise well from small data, so we should use one for the ten-million-row problem too." The property that makes them good at small n is the quadratic programme that makes them impossible at large n. Different tool.
"The decision value is 2.3, so it is a confident defect." It is 2.3 margin-widths from the boundary. Confidence needs a calibration step that was never done.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

MODEL-SPECIFICThe coupling of γ to feature scale is a kernel-method property; tree ensembles on the same forty features are scale-invariant and would not have the unscaled-kernel failure, though they would have their own on extrapolation beyond the training range.
SCALE-SPECIFICKernel SVMs are affordable to a few tens of thousands of rows and become the wrong tool past a few hundred thousand; linear SVMs scale to millions but give up the curved boundary that was the reason to choose the method.
DATA-SPECIFICOn a few thousand dense, clean numeric rows an RBF SVM is often the strongest model available; on sparse high-cardinality categorical data a linear model is usually better, and on raw images or text a learned representation replaces the kernel entirely.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringfeature-pipelines

Domains that do not exist yet

— Testing & Reliability Engineering — the scaler statistics shipped in the artifact and the per-feature drift thresholds are a contract between the sensor pipeline and the model; keeping that contract tested when either side changes is a reliability discipline this domain names but does not own.