Learn Machine Learning Engineering

Not "which algorithm". Turning data into a system that learns useful patterns, generalises, serves predictions and stays measurable after it ships — and knowing which neighbouring domain owns each thing it depends on.

Raw data → dataset → features → split → model → training → evaluation → artifact → deployment → inference → feedback. Eleven stages, each a place an assumption enters, and the loop back is what makes it a system rather than a script.

Sixteen failure classes, each entering at a specific pipeline stage, most invisible to the offline metric. Learning to name them by stage is the difference between debugging a model and retraining it in the dark.

A program encodes rules someone wrote. A model encodes patterns from data it was shown — and therefore inherits the data's biases, gaps and timing. The model is a set of assumptions with weights attached.

Problem → Target → Data → Representation → Split → Model → Training → Evaluation → Validation → Deployment → Inference → Monitoring → Drift → Retraining. Fourteen questions in order, and the order is the method.

Libraries hide optimisation, AutoML hides search, feature stores hide synchronisation, model servers hide inference, cloud platforms hide infrastructure, foundation models hide training. Use all of them — and know what each one is hiding when it breaks.

"Users are cancelling subscriptions" is a situation, not a task. Six questions turn it into Input X → Model → Prediction ŷ → Decision, and each one skipped is a model that answers something nobody asked.

The model exists to change a decision. Name the decision, its owner, its capacity and its moment first, and most model choices — target, features, metric, threshold, inference mode — are made for you.

`P(churn) = 0.78` is a prediction. "Offer a retention discount?" is a decision. The model produces the first; a threshold, a cost and a policy turn it into the second, and none of those three lives in the model.

The target must encode the outcome you actually care about, at a horizon, from a moment. `churned = cancelled within 30 days of the snapshot` is a target; `churned` is not.

Labels are built, not found. A versioned, tested query over raw events, parameterised by the snapshot moment and the horizon, is the difference between a label and a column that happened to be there.

A feature that carries the answer — `cancelled_at` used to predict `will_cancel` — gives excellent offline metrics and an invalid model. Leakage is about when information exists, not which columns are forbidden.

A rule works; labels cannot be observed; the decision cannot use a probability; a wrong prediction has unbounded cost; the data does not exist at prediction time. Any one of these is a reason to stop, and the formulation is where you find out.

A labelled target turns learning into function fitting. The model is only as right as the label, and the label was made by a process nobody wrote down.

No labels, so no loss against the truth. The model finds structure in whatever the features and the distance say — and nobody checked that those mean anything to the business.

A few thousand labels and a few million unlabelled rows. The unlabelled data helps exactly when it comes from the same distribution as the labels — and that is the thing you cannot check with labels.

The data labels itself: hide part of it and predict it back. The signal is free and abundant, which is why it works — and why the model learns whatever the corpus contains, including what you did not want.

Every paradigm is defined by where the gradient's target comes from. That source decides what the model can be wrong about without anyone noticing.

Have labels? Can you get them? What do they cost, and how long until they arrive? What does "structure" mean to the business? The paradigm is the answer to those questions, not a preference.

The output is a number. The loss decides which errors that number is allowed to make, and the business rarely agrees with squared error about which errors are expensive.

The model outputs a probability; the product needs a decision. The threshold between them is where the business cost lives, and it is the part that gets defaulted to 0.5.

The output is an order, judged by what sits at the top. The label is usually a click, which was produced by the previous ranking — so the model learns the old order as much as relevance.

k-means and hierarchical clustering find groups under a distance you chose. A visually separated cluster is a fact about the geometry, not about the business — until something external says otherwise.

PCA keeps variance; UMAP and t-SNE keep neighbourhoods, approximately. Neither keeps meaning, and a 2D picture of a 300-dimensional space is a drawing, not a map.

The model ranks how unusual each point is. Unusual is not the same as bad, positives are rare, and someone has to read the top of the list — so precision there is the whole product.

Raw data becomes a dataset through filtering, joining, labelling and feature creation. Each stage is a decision, and each decision can introduce bias or leakage that no model can undo.

One row is one user, or one transaction, or one user-day, or one query-document pair. Choosing the grain decides the snapshot date, the label window, and what counts as a duplicate.

Random, stratified, temporal and group-based sampling each preserve a different property of the population. Which property matters depends on what the model will meet in production.

The dataset only contains the cases that reached the step where the label was recorded. Approved loans have repayment labels; declined ones do not. The model learns about the selected, and is deployed on everyone.

The table contains the customers, companies or machines that are still here. The ones that failed were deleted, archived or never joined, and the model learns what survivors look like.

When positives are one in a thousand, always predicting negative is almost perfectly accurate and completely useless. Imbalance decides the metric, the split, the threshold, and whether the probabilities can be trusted.

The label is the thing the model is trained to reproduce. Noisy, delayed, drifting, disputed or machine-generated labels put a ceiling on everything downstream, and the ceiling is invisible in the metric.

Three sets with three jobs: learn parameters, choose between models, and estimate final performance once. The percentages are a consequence of the jobs, not a rule.

Shuffle the rows and cut. Correct when rows are independent and production looks like the training period. Wrong, and optimistic, whenever time or repeated entities are in the data.

Train on the past, validate on the future. The only split that measures the thing production actually asks for — how well the model generalises to a period it did not see.

When the same entity appears in many rows, all of its rows go to one side of the split. Otherwise the model is evaluated on recognising entities it already saw, and production is full of entities it has not.

When positives are rare, a plain random cut can leave validation with too few of them to say anything. Stratifying fixes the class ratio per set so every fold holds a known number of positives.

Four questions decide the split: is there time in the data, do entities recur, are positives rare, and will production see new entities or a new period? The answers compose into one strategy.

Leakage is information from the answer reaching the model during training through a route that will not exist at prediction time. The offline metric improves; the product does not.

A feature that is derived from, caused by, or written by the same process as the label. It looks like a column; it is the answer.

Information from after the prediction time reaches the features: a future timestamp, a window that crosses the snapshot, a random split of time-ordered data.

The same user, patient or device appears on both sides of the split. The model memorises the entity, the evaluation rewards it, and production is full of strangers.

A scaler, imputer, encoder or feature selector fitted on the full dataset before the split has seen the validation rows. The order of operations is the leak.

The data is clean and the pipeline is ordered correctly. The leak is the engineer: tuning on the test set, peeking repeatedly, picking the best of many runs on one holdout.

A checklist run on every feature before the offline number is believed: when is it computed, from what, is it available at prediction time, does it correlate suspiciously, is it near-perfect on a subgroup.

A feature is a transformation from raw records to a number the model can use. It is learned from training data, and it has to be reproduced identically at serving time — which is where it usually breaks.

Per-entity counts, sums, rates and recency over windows. They dominate tabular models, and they are the main source of train/serve skew because they depend on a clock, a source and a null policy at once.

Bucket edges, means and standard deviations are fitted on the training fold and shipped with the model. Refit them anywhere else and the model receives inputs from a transformation it never learned.

One-hot, ordinal and embedding encodings turn categories into numbers. Each has a vocabulary that was fitted on training data, an unseen-category policy, and a serving path that must apply both identically.

Replace a category with the mean label for that category. Powerful on high-cardinality features, and a leak unless the rate for each row is computed without that row, out of fold, with a smoothed prior.

Windows, lags, recency, calendar features and "as of" timestamps. Every one is anchored to a clock, and the rule is that the anchor is the prediction time and no window ends after it.

Why a value is missing is information. Imputation is fitted on the training fold and shipped, the missingness indicator is often the better feature, and the null policy must be identical at serving.

Either a person decides what the model sees, or the model decides. Each choice hides something, and the learned one ships inside the artifact and must be versioned like weights.

Fewer features means fewer serving dependencies, less leakage surface, and a smaller lie when the selection is done outside the training fold — which is where it is usually done.

Split gain, coefficients and every other model-specific importance answer one question: what does this model use? They do not answer what matters in the world, and unscaled coefficients do not even answer the first one.

Shuffle one column, re-score the model on held-out data, and the drop is what the deployed model depends on. Done on training data it measures memorisation; done one correlated feature at a time it splits the credit and hides the group.

Every importance and attribution method describes how a model's output depends on its inputs. None describes what would happen if you changed the world. "X predicts Y" and "X causes Y" are different claims, and the business hears the second.

A metric with nothing to compare it to is a number, not a result. Before anything complex: a rule, a constant predictor, a linear model, a shallow tree — and the question of whether the proposed model beats a useful one by enough to pay for itself.

The heuristic the business already uses is the strongest baseline most models face, and the honest reason a model has to win by a margin: the rule is free to run, already trusted, and already in production.

The constant predictor is the floor of every metric. Under imbalance it wins accuracy without looking at a single feature, and for regression it defines R² = 0 — which is why scoring it first is how you find out whether the metric means anything.

Logistic or linear regression as the first real model: cheap to fit, cheap to serve, readable, and the reference for everything after it. If the complex model cannot beat it clearly, the complexity is not earning anything.

A comparison is only a comparison on the same split, the same metric, the same threshold policy, with an interval — and the margin has to be read against what the winner costs to serve. A small gain that costs a GPU and eighty milliseconds is a loss.

ŷ = w·x + b. A weight per feature, a bias, a loss that says which mistakes hurt — and a set of assumptions the weights only make sense under.

The residual plot is the diagnostic; the single metric is the summary. Heteroscedasticity, extrapolation and unscaled coefficients are all visible there and invisible in the RMSE.

Linear score → sigmoid → probability, trained by gradient descent on the log loss. The threshold that turns the probability into a decision is a different step, owned by someone else.

The sigmoid turns a score into a number between 0 and 1. Whether that number is a probability is a fact about calibration on the deployment distribution, not about the function.

The threshold is not part of the model. It is the point where a business decision about costs and capacity is written down, and it deserves an owner, a config and a review.

L1 makes weights zero, L2 makes them small, elastic net does both — and all of them penalise a feature in proportion to its scale, so the scaler is part of the model.

Four counts — TP, FN, FP, TN — and four business outcomes with four different prices. Every classification metric is a way of reading this table; read the table first.

Precision reads the flagged column: how many alarms were real. Recall reads the positive row: how many real cases were caught. F1 averages them as if the two mistakes cost the same, which they never do.

Flag when P × cost_FN exceeds (1 − P) × cost_FP. The threshold falls out of the costs and the calibrated probability; 0.5 is what you get when the costs are equal and nobody checked.

When the positive class is one in a thousand, predicting "no" every time is 99.9% accurate. Accuracy measures the majority class; use the metrics that read the positive row and the flagged column.

The probability that a random positive scores above a random negative. A pure ranking metric — invariant to threshold, to prevalence, and therefore blind to the precision that prevalence destroys.

Precision against recall across every threshold, and the area under it. It follows the positive class, so it falls when the flagged set fills with negatives — which is exactly what ROC AUC cannot see.

Does 0.8 mean 80%? The reliability curve answers it bin by bin. Calibration matters when the probability is multiplied by a value; it matters not at all for a pure ranking.

Squared error punishes a large miss quadratically and answers in squared units; RMSE restores the units but keeps the outlier sensitivity; MAE is the median-like metric that treats every unit of error the same.

R² is the fraction of variance the model explains relative to predicting the mean. It can be negative out of sample, it is not comparable across datasets, and a high value can describe a model that is useless for the decision.

Mean absolute percentage error reads naturally and fails badly: undefined at zero, dominated by small actuals, and asymmetric between over- and under-forecasting. On a demand forecast it blows up on exactly the low-volume items nobody was worried about.

Derive the metric from the cost of being wrong: is a ten-unit miss the same on a hundred-unit item as on a ten-unit item, are large misses catastrophic or merely bad, and is the decision actually a threshold on the forecast — in which case it is classification in disguise.

A model metric describes the model; a business metric describes what happened when the model's output was acted on. The map between them is the operating point and the action, and a better model metric can produce a worse business outcome.

Offline evaluation scores a model on a historical dataset produced by the previous policy. Online evaluation measures what happens when the model acts on live traffic. Strong offline numbers are a reason to run the online test, not a substitute for it.

k-fold cross-validation trades k trainings for a lower-variance estimate and a spread. It is the right tool for small data and model selection, the wrong tool for temporal or grouped data unless the folds respect the structure, and it is not an evaluation of the model you will ship.

When the model will predict the future, validate on the future: forward-chaining folds, a gap between training end and validation start equal to the label delay, and never a shuffle. A random split on temporal data is a leakage simulator with a nicer name.

An aggregate metric is a weighted average over subgroups, and the weights are the dataset's, not the business's. A model can improve on average and regress on the segment that matters, and only a sliced evaluation can see it.

A validation metric is a sample statistic with an interval around it. Two models compared on the same holdout need a paired comparison; a small test set cannot distinguish small improvements; and every comparison made against one holdout erodes it a little.

The test set is touched once. Every look costs information; hyperparameter search, feature selection, early stopping and model selection all happen on validation. A team that picks the best of forty runs on the test set has shipped noise with a certificate.

Every model is wrong in two ways at once: too simple to represent the pattern, or too flexible to ignore the noise. The gap between training and validation error tells you which.

A model that memorises the noise in its training set scores perfectly on that set and poorly on the next one. Every route to a better training number is also a route to this.

A model with too little capacity, or the wrong representation, misses structure that is plainly in the data. It is the honest failure — visible offline — and still the one most often fixed with the wrong tool.

Error against training-set size, for training and validation together. The shape says whether more data, more capacity or better features is the fix — before any of them is tried.

Every way of refusing part of the training fit: L1, L2, dropout, depth limits, shrinkage, early stopping. The strength is a hyperparameter, it is tuned on validation, and for the penalty forms the features must be on one scale.

Stop training when validation loss stops improving, keep the best checkpoint, and accept that the validation set you stopped on is no longer an unbiased estimate of anything.

A tree is a sequence of `feature < threshold?` questions ending in a constant. Readable, axis-aligned, piecewise flat — and incapable of predicting a value it has never seen.

Gini, entropy or variance reduction score each candidate cut; the greedy search picks the best one at each node and recurses. Stopping rules are the only thing between that and a leaf per point.

Many deep trees, each on a bootstrap sample and a random feature subset, averaged. Variance falls because the trees disagree; the out-of-bag rows give a free validation estimate; the artifact is large.

Fit a tree, compute the residuals, fit the next tree to them, repeat. Each tree follows the negative gradient of the loss; the learning rate shrinks each step; the number of trees is the capacity knob, chosen by early stopping.

Second-order gradients, a regularised objective, histogram binning, leaf-wise growth, native missing-value and categorical handling — what the fast implementations add to boosting, at the level of the mechanism and never the API.

A single tree, a forest, boosting and a linear model scored on quality, latency, cost, interpretability, data needed and operations — and the cases where boosting is the wrong answer even though it would win the benchmark.

No training, all the cost at inference, and a distance that is only meaningful in a scaled space. The mental model behind every embedding retrieval system.

Count the evidence per class and multiply, pretending every feature is independent. Wrong about the world, surprisingly right about text, and confidently miscalibrated.

The widest street between the classes, defined by the few points closest to it. Kernels make a linear boundary curve; scaling is mandatory; the cost grows badly with the number of rows.

The interview question whose red-flag answer is a product name. A strong answer asks about the data, the latency, the interpretability, the metric and the cost before naming anything.

Input → linear layer → activation → hidden layers → output. A stack of learnable linear maps with nonlinearities between them, trained by gradient descent on a loss. Not always better.

z = w·x + b, then activation(z). One unit is a logistic regression; a layer is many of them sharing an input; the network is the same thing composed.

ReLU, sigmoid, tanh, GELU. Nonlinearity is what stops a stack of linear layers collapsing into one; the choice decides which gradients survive the trip back.

Input → layers → prediction, as a sequence of matrix multiplications with a batch dimension. This is where the FLOPs go, and where inference cost is decided.

Prediction vs target → loss. MSE, binary and categorical cross-entropy, ranking losses. The loss is what the optimiser minimises; it is not the metric the business cares about, and the gap is the design.

Forward pass → loss → backward pass → gradients → parameter update. The chain rule applied node by node, in reverse topological order, on the 2-2-1 network the lab runs.

Nodes are operations, edges carry values forward and gradients backward. Reverse mode is cheap for many parameters and one loss, the framework builds the graph as you call it, and the activations it stores are the memory bill.

θ ← θ − η∇L. One update rule, one number that decides whether it crawls, converges or explodes. The learning rate is the single most important hyperparameter in deep learning.

Three ways to decide what multiplies the gradient. None is universally best: Adam converges fast and sometimes generalises worse; SGD with momentum is still the default in much of vision; the choice interacts with the learning rate and with weight decay.

An epoch is one pass over the data. A batch is the subset used for one gradient estimate. A step is one parameter update. "We trained for ten epochs" says nothing until you know the batch size.

Batch size trades memory, throughput and gradient noise against each other, and it moves the right learning rate with it. Larger batches want larger rates — up to a point — and small batches regularise for free.

Backpropagation multiplies one Jacobian per layer. A chain of factors below one shrinks the gradient to nothing by the early layers; a chain above one blows it up. Depth is hard to optimise for this reason, and every remedy attacks the product.

Batch norm normalises each feature over the batch; layer norm normalises each example over its features. The difference decides whether the layer behaves the same at training and inference — and batch norm does not, which makes it a train/serve skew source with a name.

Where the weights start decides whether training can begin; the warm-up and decay decide how it ends. A loss curve that plateaus, diverges or oscillates is a report on those choices, and a seed is not a reproducibility strategy.

A discrete entity — word, user, product, document — becomes a dense vector that is a parameter of some model, learned on a proxy task. The geometry encodes what that task rewarded, not "meaning".

Lookup tables as parameters, a contrastive signal from observed pairs against sampled negatives, and the consequence: rare entities get noise, and the table is part of the model artifact and must be versioned with it.

The dot product divided by the norms: the angle between two vectors, ignoring their length. Right when magnitude is noise, wrong when magnitude is signal — and at scale, nearest neighbours are an index problem, not a formula.

A 2D plot of high-dimensional vectors is a lossy projection. PCA keeps variance, not neighbourhoods; t-SNE and UMAP keep local structure and invent global structure. The clusters, distances and neighbours in the picture are not the ones the model uses.

Retraining an embedding model produces a new coordinate system. Vectors stored from the old model are incompatible with it — a version mismatch, not a quality problem — and the vocabulary and the entities drift underneath as well.

A convolution slides one small set of weights over the whole input. That weight sharing is a belief about the data — the same pattern matters wherever it appears — and it is the reason a CNN needs far fewer examples than a fully-connected net on pixels.

A recurrent network carries a hidden state step by step through a sequence; that is elegant and it is why long dependencies were hard. Transformers replaced the recurrence with attention so every position can be computed in parallel — and simpler models still win many forecasting problems.

Tokens become embeddings, attention mixes information across positions, a feed-forward layer transforms each position on its own, and residual connections plus normalisation let dozens of those blocks stack. Knowing where the parameters and FLOPs live is what turns "use a transformer" into a cost you can budget.

Every token asks a question (query), every token advertises what it holds (key), and each token's new representation is a softmax-weighted mix of what the relevant tokens carry (value). The formula fits on one line; the weights it produces are a computation, not an explanation.

Attention is a weighted sum over a set: shuffle the tokens and it computes the same thing. Order has to be injected explicitly — learned, sinusoidal, relative or rotary — and the scheme you pick decides whether the model can say anything sensible past the lengths it was trained on.

Encoder-only models turn text into a representation and are what you want for classification and embeddings; decoder-only models generate the next token; encoder–decoder models read one sequence and write another. Pick by what the output must be, not by which is newest.

A foundation model is pretrained once on broad data and reused across many tasks. "Pretrained" hides a training system you did not run, data you did not choose and an objective you did not pick — and what you inherit shows up as serving cost and as benchmark numbers that are not your task's numbers.

Pretrained model + task data → adapted model. The early layers carry structure that transfers; the late layers carry the old task. Freeze what transfers, train what does not, and expect the labelled-data requirement — and the split arithmetic — to change.

Fine-tuning changes the weights with your labelled data; prompting and retrieval change the inputs and leave the weights alone. The first is an ML Engineering job with training, evaluation and a new artifact; the second is Agentic Engineering. Knowing which one you need is most of the decision.

Keep the base frozen, train a small number of new parameters — an adapter, a low-rank update to a few weight matrices — and ship the delta. Many task adapters can share one base in memory, which changes what an artifact is and what serving looks like; the price is a quality ceiling the base sets.

Parameters are learned from the data. Hyperparameters are chosen before training, judged on validation, and belong in the experiment record — because they decide what the learning is allowed to do.

Grid search is exhaustive and exponential. Random search covers each important setting better per trial, because most settings turn out not to matter. Neither is allowed anywhere near the test set.

When a trial costs hours, spend the trials sequentially: model the objective from the trials so far, choose the next one to balance exploration and exploitation, and kill trials that are clearly losing before they finish.

A search costs compute, time and validation-set credibility, and returns less with every trial. Tune the learning rate first, stop when the curve flattens, and remember that a fixed leak or a better feature usually beats any amount of tuning.

AutoML runs a search over pipelines and hyperparameters against a validation metric and hands you the winner. What it hides is the search space, the preprocessing leakage it may have committed inside the loop, the validation set it has now overfitted, and the serving cost of the pipeline it chose.

The target is a future value of the series you already have. Features are lags and windows that end at the forecast origin, the naive forecast is the baseline, and beating it is harder than it looks.

Most series are a level, a trend, one or more seasonal cycles and calendar effects on top of noise. Model each explicitly or difference it away — and know that a model fitted to one regime is a bet that the trend continues.

One step ahead and twelve steps ahead are different problems with different errors. Direct and recursive strategies trade compounding error against training cost — and the horizon that matters is the one the decision needs.

Forecast the series, compare the actual to the forecast, and flag when the residual leaves a band. The band is a threshold with a false-alarm cost, the baseline must know about seasonality, labels are scarce, and an alert with no owner is noise.

Move the origin forward through time and score each forecast against what happened next; never shuffle. Report MAE or RMSE scaled against the naive forecast, per horizon and per segment — and treat MAPE with suspicion near zero.

A recommender is a loop, not a model: events feed candidate generation, candidates are ranked, the ranking decides what users see, and what users see decides the next batch of events.

Learn from who interacted with what, with no item attributes at all — and inherit every bias in who was shown what, because the missing entries in the matrix are not negatives.

Recommend from what items and users are, not from who touched what. It works on day one for a new item and is limited to what the attributes can express.

Millions of items cannot be scored by a rich model inside a page-load budget. Retrieval narrows to hundreds with a cheap model; ranking orders them with an expensive one; each stage has its own metric and its own way to fail.

A new user or item has no interaction history, so an interaction-trained model has nothing but noise for it. The answers are popularity, content, asking, and deliberately showing it — none of which is a better model.

The model decides what users see, what users see decides what they click, and what they click is the next training set. Retraining on that log does not correct the loop — it tightens it.

Showing the best-known item earns the most today and learns the least. Some exploration is the price of data you can trust — and in some domains that price cannot be paid.

Every run records the code, the data, the features, the configuration, the metrics, the artifacts and the environment. A number without that record is a claim nobody can check.

Same code, same data, same seed, different number. Reproducibility is a property of the entire environment — kernels, data order, library versions, reduction order across workers — and a seed pins only one of them.

A seed fixes which random draws the code makes — the split, the initial weights, the shuffle, the dropout masks. Each is a different seed with a different effect, and the variance across them is a number a good comparison reports.

A table name is not a version. A dataset the model trained on must be an immutable snapshot with an identifier that resolves to the same rows forever — or the run record points at nothing.

A feature name is a contract whose definition changes; a model artifact is a file whose meaning depends on which definition it was trained against. The two versions must travel together, and a mismatch is a production failure with no error message.

Raw data → Dataset v12 → Features v7 → Training Run 482 → Model v19 → Production. The graph that answers "which data did the production model learn from" during an incident, recorded by machines rather than remembered by people.

A weights file alone is not a model. The artifact is parameters, architecture, fitted preprocessing, feature order, version and training metadata — and serving needs all of it.

The registry is a state machine over artifacts — Candidate, Registry, Staging, Production, Archived — that stores lineage, metrics, approvals and the feature-definition version, so "which model is live" has one answer.

A challenger is promoted on quality, latency, memory, cost, robustness and — where relevant — fairness, compared against the champion on the same slice with the same threshold policy. One improved offline number is not a reason to ship.

Hashes and signatures prove the bytes serving loads are the bytes that were evaluated. Deserialisation formats that execute code on load, and a serving process that loads the wrong file, are the two ways that proof gets skipped.

The normaliser's means and standard deviations, the encoder's vocabulary, the imputation values, the feature order and the threshold are fitted on the training fold and ship with the weights. Recomputing any of them at serving time is a different model.

Dataset → Model → Predictions, on a schedule. When the prediction can be precomputed, batch is the cheapest and most debuggable mode — and the staleness window is a property to design, not a defect.

Request → Features → Model → Prediction → Response, inside a latency budget. The feature fetch is usually the latency, and the timeout and fallback are part of the model's quality, not an infrastructure detail.

Continuous events drive predictions: the model sits inside a stream processor, features are state kept per key, and ordering, late events and where the model sits in the topology decide correctness more than the weights do.

How fresh must the prediction be, can it be precomputed, does it need live features, what is the latency budget, what is the volume — those five questions decide batch, online, streaming or hybrid. Online is often unnecessary, and the churn case shows why.

Client → Backend → Model Service → Artifact → Prediction. Where preprocessing runs, model-in-process against model-as-service, versioned endpoints, warm-up and health checks — and the line where the Backend domain takes over.

Individual requests are grouped into a batch before the accelerator sees them. Throughput rises because the hardware runs one large matrix multiply instead of many small ones; latency rises because every request waits for the batch. Dynamic batching with a maximum wait is the knob.

A workload decision: model size, available batch size, latency budget, cost per prediction and utilisation. A small tree model on CPU beats a GPU round-trip; a large transformer at volume does not fit on CPU. "GPU makes inference faster" is false as stated.

The weights are identical in training and production. The features are not. A model can be exactly right about inputs it will never see again.

A feature store is optional infrastructure that makes one feature definition serve both training and low-latency inference, with lineage attached. It is one answer to skew, not a prerequisite for ML.

A training example at time T may only use information that existed at T. The as-of join is how you build that, and the offline store exists to make it cheap.

Features update in seconds, minutes, hours or days. The model was trained on values of a particular age, and the serving architecture must deliver the same age or the model is reading a different signal.

A prediction request is parsing, feature fetch, preprocessing, model compute, postprocessing and network. The model is rarely the slow part for tabular systems, and almost always is for large networks.

Throughput is predictions per unit time; latency is how long one waits. Batching raises the first by spending the second, and queue depth — not CPU — is the signal that says you are running out of both.

When the model or its features are unavailable, the system must return something defined: the previous model, a rule, a cached score, a default ranking, or an explicit "no prediction". Which one is a product decision.

A GPU is thousands of simple cores doing the same matrix arithmetic in lockstep. It is fast only when there is enough parallel work to fill it, which is why a single small request leaves it mostly idle.

What fits on the device is parameters times bytes per parameter, plus activations, plus — for training — optimizer state. What runs fast is bounded by how quickly those bytes can be read, and for large models every token reads all the weights.

Storing weights in fewer bits — FP32 to FP16, BF16, INT8 — shrinks memory and speeds up memory-bound inference. The quality cost is real, concentrated on rare inputs, and only visible if you evaluate on the same slices you used before.

Quantization, pruning and distillation are three different bargains: fewer bits per weight, fewer weights, or a smaller model taught by the larger one. They trade quality, latency, cost and engineering effort differently, and can be combined.

Pruning removes weights, and only speeds things up when it removes them in shapes the hardware can skip. Distillation trains a small model on a large model's outputs, and inherits everything the large model believed.

Cost per prediction is hardware cost per hour divided by predictions per hour, plus feature fetch and storage. Utilisation is the lever, and the first question is whether the prediction needs this model at all.

Splitting a training run across many devices buys compute and pays in coordination. It is needed when the data, the model or the calendar does not fit on one machine — and for most models it is neither needed nor free.

Every worker holds the full model and a different shard of the data; each computes gradients on its shard and the gradients are averaged. It is the simplest split, and it silently multiplies the batch size.

When the model itself does not fit on one device, you cut the model rather than the data — across layers, inside matrix multiplies, or across the optimizer state. Every cut moves activations or weights over the wire, and the wire becomes the bottleneck.

Workers agree on a gradient by all-reduce — a ring exchange that is bandwidth-optimal — and the choice between waiting for everyone and not waiting decides staleness, straggler exposure, and whether two runs can ever produce the same bits.

A training checkpoint saves model weights, optimizer state, the step counter, the data position and the RNG state so a run can resume exactly where it died. It is not the model artifact, and a resume that does not restore all of it silently trains a different run.

A training run costs GPU-hours, CPU-hours, storage, network and — the multiplier that dominates — the number of times you run it. A hyperparameter search turns one run's cost into a bill, and most of the questions that reduce it are not about the hardware.

Data tests, feature tests, training tests, model tests, serving tests, integration tests and drift tests — seven layers because a model can fail at every one of them while every unit test stays green.

Schema, nulls, ranges, cardinality, target prevalence and distribution against the training reference — and the one test that catches most leakage: no feature timestamp may exceed its prediction time.

On a tiny dataset, in CI, in minutes: the pipeline runs end to end, the loss goes down, an artifact appears, and the model can memorise a handful of examples. A pipeline that cannot overfit ten rows is broken, whatever the full run reports.

A probability is in [0, 1]. No output is NaN. A higher income does not lower a credit score. A change to an irrelevant field does not change the prediction. Invariants are the tests a model must pass regardless of its metric, and the ones a metric cannot express.

Request schema, preprocessing equivalence, model version, output schema, latency budget and fallback behaviour — the contract between the artifact and the request path, tested on every deploy, with a replay against the training path as the test that catches skew.

Missing features, extreme values, noise, rare segments and corrupted inputs — the test is not whether the model stays accurate under damage but whether it does what the design says it should: degrade gracefully, refuse, or fall back.

A challenger with a better aggregate metric can still lose the cases the champion passes. A regression test holds the challenger to the champion's slices and to a golden set of known hard examples — and the golden set is a leakage risk the moment anyone trains on it.

One aggregate number hides that a model can be a different model for different groups. There is no single fairness metric to optimise; choosing one is a policy decision, and several of them cannot all hold at once.

An explanation describes the model, not the world, and describes a wrong model just as fluently. Know whether you need a global picture, a local reason or a counterfactual, and whether an interpretable model would make the question go away.

A model that predicts Y from X has learned that X and Y move together in data generated by an old policy. Acting on X to change Y is a different question, and usually needs an experiment rather than a model.

A training set is personal data, an artifact can memorise it, and a prediction log is a record of people. Privacy is a design property of the pipeline — minimisation, retention, access, and honest limits on anonymisation.

A human in the loop is a threshold, a queue, and a source of labels. Decide where the human decides, size the queue from the threshold, watch for automation bias, and remember that overrides are training data — and biased training data.

A model needs everything a service needs, plus three distributions a service does not have: features in, predictions out, and outcomes back. Four layers, each with an owner, each catching a different failure.

The input distribution changed. A distance metric between the training reference and this week's traffic says so on the day; whether it matters depends on where the inputs moved to, and that needs the outcomes.

One feature's distribution moved. Before it is drift it might be a bug: a null-rate spike, a unit change, a renamed category. Diagnose the pipeline first, because retraining on a broken feature teaches the model that broken is normal.

The output distribution moved. It is the earliest model-level signal, needs no labels, and is the one that catches train/serve skew on rollout day — because the model reacts to its inputs immediately and to the truth never.

The relationship between features and outcome changed. The inputs did not move, so no input monitor fires; the scores did not move, so no prediction monitor fires. Only the outcomes reveal it, and they arrive late.

A distribution can change legitimately and the model can handle it. "Drift means retrain" retrains a working model on the strength of an input metric, costs a training run and a rollout, and answers a question the metric never asked.

The outcome arrives weeks or months after the prediction. Every quality number on the dashboard is about the past; the architecture has to say how far past, join outcomes back by id, and use proxies honestly in the meantime.

Quality over time went down. Five different causes produce that chart, only one of them is fixed by retraining, and two are made worse by it. Diagnose in order — bug, product change, feedback loop, data drift, concept drift — before touching the model.

Retraining is a change to a running system with a cost, a risk and a precondition. Four questions decide whether it is due; "drift" is not one of them.

Scheduled, drift-triggered, performance-triggered, manual and continuous: each is right for a particular ratio of label speed to world speed, and continuous training has risks the others do not.

A candidate earns production by beating the incumbent on the same traffic under the same threshold policy, on more than one number. A better validation score is a nomination, not a promotion.

The candidate scores production inputs but controls nothing. It catches skew, latency and crashes before a user sees them — and it cannot measure business impact, because it never makes a decision.

Give the candidate 1% of decisions, then 5%, 25%, 100%, watching quality, latency, cost and errors at each step — with the honesty that a 30-day label makes a 30-day canary.

The only measurement of business impact is to let two models decide for two comparable populations and compare what happens — with stable assignment, guardrails, enough sample, and honesty about interference and about the users in the experiment.

When the model is wrong, go back; when the model is gone, degrade. Rollback must restore the feature definition with the artifact or it reintroduces skew, and the previous model has to be warm.

A healthy model server can serve a wrong model indefinitely. Observability for a model means tracing Request → Features → Model Version → Prediction → Decision → Outcome, and watching signals service health does not have.

The prediction log is the monitor's input, the incident's evidence, the outcome join's left side and the next training set. Log what is necessary, reference what is sensitive, and decide retention before the first row.

One request id, from the click in the frontend through the backend, the feature service, the model server and the decision, to the outcome event weeks later. Each hop records something specific, and a hop that drops the id is where the next incident becomes a guess.

Conversion fell after a deploy. Investigate from the business metric down — prediction distribution, model version, feature values, feature pipeline, raw data — in that order, and do not retrain until the cause has a name.

A model incident postmortem records the assumption that broke, the signal that should have fired, the label delay that hid it and the test now added — and never concludes that "the model" was at fault.

Engineering practices and platform capabilities that make ML systems reproducible, testable, deployable, observable and maintainable. Not a product, and not a cluster.

Data → Validation → Training → Evaluation → Artifact → Registry → Deployment → Monitoring. Each stage has a way of failing that the next stage cannot see.

Three different loops with three different triggers. CI proves the code and data are sound; CT proves a new model can be trained; CD proves it is safe to serve. A green one proves only its own claim.

Shared capabilities — dataset access, feature pipelines, training jobs, tracking, registry, serving, monitoring, GPU scheduling — built once for many model teams. Premature before the third model.

Every managed ML service is an implementation of a primitive you should already be able to name — a training job, a GPU, a registry, a hosted endpoint. Learn the primitive, then map the vendor.

A DAG of data → features → train → evaluate → register → deploy, with scheduling, retries and backfills. The ML-specific hazards: a training step that is not idempotent, an evaluation gate, and artifact promotion as a step.

Before buying cheaper GPUs, ask whether a simpler model works, whether training can happen less often, whether inference can be batched, quantized, cached or precomputed. Utilisation is the lever.

Six ways an ML system is exposed that a service is not: poisoned training data, tampered artifacts, memorised sensitive data, adversarial inputs, an untrusted supply chain and abusable inference. Defensively, at the level of what to trust.

A corrupt training source teaches the model wrong behaviour, and the offline metric — computed on the same corrupt data — approves. Provenance, validation, trusted pipelines and slice evaluation are the defence.

Small, deliberate changes to a valid input flip the prediction. Defensively: validate content, test robustness, use ensembles and monitor confidence — and accept that fraud and spam are adversarial by nature.

Pretrained weights and public datasets are dependencies: unpinned, unhashed, unsigned, and loaded by formats that execute code. Pin, hash, sign, use safe formats, and record provenance in the registry.

An endpoint that answers anyone reveals its decision function, its training data and its cost structure. Authenticate, rate-limit, return decisions rather than probabilities, and bound spend on GPU endpoints.

Sources → data platform → features → training → registry → serving → application → monitoring. Eight boxes, five owning teams, and four interfaces that decide whether the system can be reasoned about at all.

Ten questions — target, latency, batch or online, freshness, volume, label delay, model size, fallback, retraining, cost — each of which decides a part of the architecture before any model is chosen.

Events → candidate generation → features and embeddings → ranking → serving → feedback. A two-stage latency budget, a loop in which the model writes its own training data, and an offline metric that measures agreement with the previous policy.

Rare positives, a strict online latency budget, asymmetric costs, an adversary who adapts to the model, and labels that arrive ninety days late. Every constraint in the domain at once.

A weekly call list for a team of fixed capacity. Batch scoring, a threshold that is a queue size, labels a month late, explanations the callers can use — and a demonstration of why the online endpoint someone will propose is unnecessary.

Query → candidate retrieval → ranking model → results. Lexical and embedding retrieval, learning-to-rank from click labels that carry position bias, NDCG-style evaluation at concept level, a latency budget per stage, and interleaving for the online test.

ML Engineering owns the model: training, fine-tuning, evaluation, embeddings, inference, serving, drift, MLOps. Agentic Engineering owns what is built on top: prompting, RAG, tools, memory, agent architecture, agent evals. The line is where the depth lives, not where the LLM is.