The problem says X → think Y
The searchable index of this domain. The left column is what an ML problem sounds like in a standup, a metric review or a postmortem; the right column is what to think before you open a notebook — the mechanism, not the slogan.
89 of 89 rows
| The problem says | Think |
|---|---|
| Train strong, validation weak | Overfitting. The model has enough capacity to memorise noise and idiosyncrasies of the training rows, so its training error keeps falling while its error on rows it has not seen stops falling or rises. Confirm with a learning curve, then reach for regularisation, early stopping, less capacity or more data — in whichever order the curve says.Overfitting → |
| Train + validation weak | Underfitting. The model cannot represent the pattern at all, so it is wrong on the rows it trained on as well as on new ones; no amount of regularisation helps, because the problem is too little capacity or too little signal in the features. Check the features and the baseline before adding capacity — a weak feature set underfits regardless of model.Underfitting → |
| Rare positive class | Precision / Recall / PR AUC. With a few positives per thousand, accuracy and even ROC AUC are dominated by the negatives and can look fine while the model finds almost none of the cases you care about. Precision and recall are computed on the positive class only, and PR AUC summarises the trade-off between them where the negatives cannot inflate it.PR AUC → |
| Future information in features | Leakage. A feature computed from data that did not exist at prediction time — a cancellation date, a total that includes the outcome, an aggregate over the whole history — carries the answer into the model, and the validation metric rewards it because the validation rows were built the same way. Production has no future, so the feature is missing or wrong there and the model collapses.Data Leakage → |
| Random split on temporal data | Temporal Leakage. A random split puts Tuesday in training and Monday in validation, so the model is evaluated on interpolating between days it has already seen rather than on extrapolating into a future it has not. The offline number is optimistic in proportion to how much the world changes between days. Split by time, and validate on a period after the training period.Temporal Leakage → |
| Same entity in train and test | Group Leakage. When one user, patient or device contributes rows to both sides, the model can recognise the entity rather than learn the pattern, and validation measures memorisation. If production will see new entities, the split must keep every entity on one side — a group split — or the metric describes a problem you do not have.Entity Leakage → |
| Unreliable probabilities | Calibration. A score of 0.8 is only a probability if, among all rows scored 0.8, about eighty percent are positive; boosted trees, SVMs and networks trained with class weights routinely produce ranks that are good and probabilities that are not. Check a reliability curve, and calibrate on held-out data if any downstream decision multiplies the score by a cost.Calibration → |
| Need robust baseline | Simple Model / Rule. Predict the mean, the majority class, last week's value, or the rule the operations team already uses — then measure it with the same metric and split as the candidate model. A baseline is not a formality: it is the number that turns "AUC 0.83" from a fact about a dataset into an argument about whether the model is worth its cost.Baselines Are Mandatory → |
| Tabular nonlinear problem | Tree Ensemble. On rows-and-columns data with interactions, thresholds and mixed types, a tuned random forest or gradient-boosted ensemble is usually the strongest single model without hand-built features, and it handles missing values and unscaled inputs natively. Try the linear baseline first anyway — if the ensemble only beats it by noise, the linear model is the cheaper thing to serve and explain.Tree Ensembles: When and When Not → |
| Serving differs from training | Train/Serve Skew. The weights are identical, but the features arriving at the model are computed by a different code path, from different data, at a different time — a pandas aggregate in training, a Java service in production — so the model is answering a different question. Log the serving features and diff them against a recomputation of the training features for the same requests.Train / Serve Skew → |
| Production inputs changed | Data Drift. The distribution of the features the model sees has moved away from the distribution it trained on — a new market, a new device, a changed form — and the model is being asked about regions of input space it never saw. Drift is a signal, not a verdict: measure quality on the drifted slice before deciding it is a problem.Data Drift → |
| Input-target relationship changed | Concept Drift. The features look the same but what they mean has changed — a price that used to predict churn no longer does because a competitor moved, fraud patterns adapted to the model — so the same input now deserves a different output. Feature monitoring will not see it; only labelled outcomes will, which makes ground-truth delay the constraint.Concept Drift → |
| Labels arrive weeks later | Ground-Truth Delay. Quality in production cannot be measured until the outcome is known — a chargeback, a churn, a delivery — so for weeks the only signals are feature and prediction distributions, which say nothing about correctness. Design the monitoring around proxies you can observe early, and make the retraining decision wait for real labels.Ground-Truth Delay → |
| Model offline good, production bad | Integration / Drift / Feedback. Rule out the boring causes first: a wiring bug, a feature served as null, a timeout hitting the fallback. Then skew, then drift, then whether the model's own decisions are changing the data it sees. Each has a different fingerprint — a step change on rollout day against a gradual decay — and none of them is fixed by retraining on the same pipeline.ML Incident Debugging → |
| GPU cost too high | Smaller Model / Quantization / Batch. Inference cost is set by memory bandwidth and utilisation, not by the GPU's peak FLOPs, so a request-at-a-time server wastes most of what it pays for. Batch requests, quantize to INT8, distil to a smaller model, or discover that a CPU meets the latency budget at a fraction of the price.Inference Cost → |
| Need safe model deployment | Shadow / Canary / Champion-Challenger. Run the new model on live traffic without acting on it, then act on a small slice, then compare it to the incumbent on the business metric — and have the rollback ready before the first step. An offline improvement is a reason to start this process, not a reason to skip it.Shadow Deployment → |
| "The model was great in the notebook" | The notebook measured one dataset, one split and one moment, with preprocessing fitted on everything it could see. Ask how the split was made, whether normalisation and encoding were fitted on the training fold only, and whether any feature knew the answer. A notebook number is a hypothesis about production, not a result.Offline vs Online Evaluation → |
| "Accuracy is 99.9%" | Ask what the majority class rate is. On a dataset where 99.9% of rows are negative, an illustrative 99.9% accuracy is exactly what a model that always says "no" scores, so the number does not say the model is good — it says the classes are imbalanced. Look at the confusion matrix and the positive-class metrics before saying anything.Accuracy Under Imbalance → |
| "The queue doubled after rollout" | The new model is slower per request, or larger, or its features take longer to fetch, and the serving system's throughput fell below arrival rate. Break the latency down — feature retrieval, transformation, model, post-processing — before blaming the weights, and check whether batching or a larger replica count is the fix rather than a smaller model.Latency Breakdown → |
| "Feature importance says X causes Y" | Importance says the model used X to predict Y on this data, which is a statement about the model, not the world. X may be a proxy for the real cause, a leak, or simply correlated with Y in the training window. Attribution is not causality; if the business intends to change X to move Y, that is a causal question with different tools.Attribution Is Not Causality → |
| "Should we retrain every week?" | Only if the world changes on a weekly scale, the labels arrive fast enough to measure it, and retraining is cheaper than the decay it fixes. A schedule is a reasonable default when all three hold; when they do not, it produces new models with no evidence they are better, and hides the feature bug that a drift investigation would have found.Retraining Strategies → |
| "The cluster plot looks clean" | A clean 2D picture is a property of the projection, not of the data. Neighbours in the projection are not the neighbours in the full space, clusters that look separated may overlap in dimensions the plot dropped, and a clustering algorithm always returns clusters whether or not the data has them. Validate groups against something outside the plot.Clustering → |
| "New users get nothing recommended" | Cold start. Collaborative signals need an interaction history that a new user or a new item does not have, so the model has nothing to condition on. Fall back to content-based features, popularity within the acquisition context, or a short exploration phase — and measure the new-user segment separately, because the headline metric is dominated by users who are not cold.Cold Start → |
| "The offline metric went up but retention fell" | The offline metric and the business metric are different quantities and the model optimised the one it could see. Clicks are not satisfaction; ranking loss is not retention. Either the metric was a poor proxy, or the improvement moved a slice the proxy does not weight, or the model found a way to score well that users dislike. Read the A/B test, not the validation set.Business Metrics vs Model Metrics → |
| "Training takes 3 days" | Before buying more GPUs, ask what the run is bound by: data loading, a small batch size that leaves the device idle, or genuinely the compute. Then ask whether the run needs to be one process at all — data parallelism with gradient synchronisation splits the batch across devices, checkpointing makes a 3-day run survivable, and a cost estimate says whether the 3 days are worth it.Training Cost → |
| "The model server timed out" | Whatever was served in that request was the fallback, not the model — a default score, a cached prediction, a rule — and the business metric for those requests reflects the fallback. Make sure the fallback exists, is deliberate, and is logged as a fallback so its share of traffic is visible; a timeout that silently returns 0.5 is a model outage nobody will notice.Serving Fallbacks → |
| "The data scientist owns the model, we just deploy it" | The model is one stage of a pipeline that runs from raw data to feedback, and the failures that matter enter at the other stages — the join, the split, the serving path, the label delay. Whoever deploys it needs to understand what it assumes, because the assumptions are what break in production; delegating that understanding is how a green validation number becomes an incident.Don't Delegate Understanding → |
| "We use MSE for the classifier because it trains" | The loss function is the definition of what the model is optimising, and a squared error on a probability output punishes confident mistakes far less than a cross-entropy does, so the gradients are weak exactly where the model is most wrong. Match the loss to the target type and to the cost of the mistakes, then check the metric you report is consistent with it.Loss Functions → |
| "We should build a classifier for this" | Start from the decision, not the model. What action changes when the prediction changes, when must the prediction exist, what does each kind of mistake cost, and can the outcome be observed at all so that a label exists? A surprising number of requests dissolve into a rule, a report or a threshold once those are written down.Decision Before Model → |
| "We'll just define churn as no login in 30 days" | A target definition is a product decision that the model will faithfully reproduce, including its mistakes. Thirty days catches seasonal users as churned and misses the paying user who logs in once to cancel. Write down what the label means, how it will be constructed from events, and when it becomes known — that last part decides the split.Target Definition → |
| "We don't have labels, so we'll do unsupervised" | Unsupervised methods find structure, not the structure you want; they cannot promise the groups correspond to a business concept. Check whether labels can be produced — by a heuristic, by a small annotation effort, by waiting for outcomes — before deciding the problem has no learning signal. A few hundred labels often beat an unsupervised method on the actual task.The Learning Signal → |
| "One row per transaction, so the dataset is huge" | Ask what one example is supposed to represent and whether the rows are independent. A million transactions from ten thousand users are ten thousand things, not a million, for the purpose of generalising to new users — and the model will behave that way in a group split even if the random split looked wonderful.What Is One Example? → |
| "We trained on the customers we still have" | Survivorship bias: the training set only contains entities that made it through a filter, so the model learns what survivors look like rather than what predicts survival. The same shape appears in loan models trained on approved loans and fraud models trained on caught fraud. Reconstruct the population as it was at decision time.Survivorship Bias → |
| "The labels came from the support team's notes" | Label quality bounds model quality. Labels produced by different people with different criteria at different times are noisy in ways the model will learn — and disagreement between annotators is a measurement you should take before training, because it tells you the ceiling. A model that "beats" the inter-annotator agreement has learnt the annotator, not the concept.Label Quality → |
| "We stratified the split so the classes are balanced" | Stratification preserves the class ratio across folds, which stabilises the metric on a rare class. It does not fix temporal leakage or entity leakage — a stratified random split on user rows still puts the same user on both sides. Choose the split strategy for how the model will be used, then stratify within it if the positive class is rare.Stratified Split → |
| "We normalised the whole table before splitting" | Preprocessing leakage. The mean and standard deviation used to scale the training rows were computed with the validation rows included, so information about the held-out set is in every training feature. The effect is usually small — which is why it is missed — but it can be large with target encoding or imputation. Fit every transformer on the training fold only, and ship the fitted transformer inside the artifact.Preprocessing Leakage → |
| "We target-encoded the category and the AUC jumped" | Target encoding replaces a category with the mean of the label for that category — computed on the same rows the model then trains on, it is the label leaking through a column. Compute it out-of-fold, or on a prior period, and expect the jump to shrink to something honest. If it does not shrink, the category was genuinely predictive.Target Encoding → |
| "We'll fill the missing values with the mean" | Missingness is often informative — the sensor that did not report, the field the user skipped — and mean-imputation erases it while inventing a plausible value. Add an indicator, ask why the value is missing, and check that the serving path produces the same missingness pattern as training; a feature that is never null in training and often null in production is skew.Missing Data → |
| "The residuals fan out as the prediction gets bigger" | The linear model's assumption of constant error variance is broken, which usually means the target lives on a multiplicative scale — try predicting the log — or a feature interaction the linear form cannot express. Residual plots are the cheapest diagnostic in the domain, and they say more than the R² does.Residuals & Assumptions → |
| "We use 0.5 as the threshold" | The threshold turns a probability into a decision, and the right one depends on the price of a false positive against a false negative, which is rarely one-to-one. Sweep it, plot precision and recall against it, and pick the point where the business cost is lowest — then recheck it when the base rate or the cost changes.Threshold Selection → |
| "R² is 0.9 so the forecast is good" | R² is relative to the variance of the target; a series with a strong trend gets a high R² from any model that follows the trend, including "last value". Report the error in the units the decision is made in — MAE in euros, RMSE in hours — against a naive baseline, and check how it degrades with the horizon.R² (Coefficient of Determination) → |
| "MAPE is 8%, which sounds great" | MAPE divides by the actual value, so it explodes near zero, is undefined at zero, and penalises over-forecasting more than under-forecasting on the same absolute error. On demand data with many zeros it is unusable. Prefer MAE or a scaled error, and quote MAPE only with the zeros handled explicitly.MAPE and Its Caveats → |
| "The AUC went from 0.81 to 0.82" | Ask for the error bar. On a few thousand validation rows an illustrative 0.01 difference is often within the resampling noise, and picking the best of ten runs by validation score guarantees you pick noise. Bootstrap the metric, look at the slices, and refuse to ship a difference you cannot distinguish from chance.Metric Uncertainty → |
| "We checked the test set and it was lower, so we tried a few more settings" | The test set is now a validation set and you have no test set. Every look at it that informs a choice leaks information about it into the model, and the reported number is optimistic by an amount you cannot measure. Tune on validation, touch the test set once, and if you need another look, hold out a fresh period.Never Tune on the Test Set → |
| "It works overall but fails for one country" | Slice the evaluation. A headline metric is a weighted average dominated by the majority segment, and a model can be excellent on average while being worse than the baseline on a slice that matters commercially or ethically. Report per-segment metrics with their sample sizes, and decide in advance which slices must not regress.Evaluation Slices → |
| "Loss stopped improving after epoch 3 but we trained for 50" | Everything after the validation loss bottomed out was fitting noise. Early stopping — keeping the weights from the best validation epoch — is regularisation you get for free, and it is also the cheapest tuning of the number of epochs. Make sure the "validation" it stops on is not the test set.Early Stopping → |
| "The tree is 40 levels deep and gets 100% on training" | A tree grown until every leaf is pure has memorised the training set, including its noise; held-out accuracy peaks at some depth and falls after it. Limit depth or leaf size, or grow many shallow trees and average them — that averaging is what a random forest is, and it is why the forest is hard to overfit by depth alone.Decision Trees → |
| "Just use XGBoost" | The right response is a set of questions: how much data, tabular or not, what latency budget, does anyone need to defend a coefficient, how are the probabilities used, what does it cost to serve? Boosting is often the strongest tabular model — and it is the wrong choice for a 300-row problem, an image, a 1 ms budget on a CPU, or a regulator who wants the reasoning.Which Model Should We Use? → |
| "k-NN was slow in production" | k-NN has no training cost and all of its cost at prediction time — every query compares against the stored set, and the stored set grows with the data. It is a fine baseline and a fine model for small, low-dimensional data with a good distance; at scale it needs an approximate index, at which point its simplicity is gone.k-Nearest Neighbours → |
| "The network outputs NaN after a few hundred steps" | The loss diverged: the learning rate is too large for the surface, the gradients are exploding through a deep or recurrent stack, or the inputs are unscaled and one feature dominates. Lower the learning rate, clip the gradients, normalise the inputs, check the initialisation — in that order, because the first is the usual cause.Vanishing and Exploding Gradients → |
| "We doubled the batch size and it got worse" | Batch size and learning rate are coupled: a larger batch gives a less noisy gradient and can take a larger step, so the same learning rate at twice the batch is effectively a smaller, more cautious optimiser that may settle in a worse basin. Scale the learning rate with the batch, and remember the noise of small batches is itself a regulariser.Batch Size and Learning Rate → |
| "The loss is fine but the gradients are tiny in the first layers" | Vanishing gradients: each layer multiplies the backward signal by something smaller than one, so the early layers barely learn. Normalisation layers, residual connections, ReLU-family activations and sensible initialisation exist precisely to keep the backward signal alive through depth. Check the per-layer gradient norms before changing anything else.Normalisation Layers → |
| "Cosine similarity says these two products are 0.92 similar" | Cosine measures the angle between two vectors in a space the model learnt, and 0.92 is only meaningful relative to the distribution of similarities in that space — in many embedding spaces everything is above 0.8. Compare against the neighbour distribution, and remember the space encodes whatever the training signal rewarded, not "similarity" in the abstract.Cosine Similarity → |
| "We retrained the embedding model and the search index broke" | Embeddings are only comparable within one trained space. A retrained model produces vectors with a different geometry, so anything stored under the old model — an index, a cache, a downstream classifier — is now being compared across spaces. Version the embedding model with the artifacts that depend on it, and re-embed on every change.Embedding Drift → |
| "We'll train a CNN from scratch on our 2,000 images" | Two thousand images is far too few to learn visual features from random initialisation; the model will memorise them. A pretrained network already knows edges, textures and shapes — fine-tune its last layers, or freeze it and train a linear head on its embeddings, and reserve from-scratch training for data at a very different scale.Transfer Learning → |
| "Fine-tuning the whole model needs more GPU memory than we have" | Full fine-tuning updates every parameter and stores optimizer state for each, which is several times the model's size. Parameter-efficient methods train a small set of added weights while the base stays frozen, cutting memory and producing a small deliverable — at the cost of some ceiling on tasks far from the pretraining distribution.Parameter-Efficient Fine-Tuning → |
| "Attention can see the whole sequence, so order doesn't matter" | Self-attention is permutation-invariant by construction: without positional information, "dog bites man" and "man bites dog" produce the same representation. Positional encodings or embeddings are what re-introduce order, and their design decides how the model behaves on sequences longer than any it trained on.Positional Information → |
| "We ran a grid over 6 hyperparameters" | A grid over six axes at five values each is fifteen thousand runs, most of them wasted on the four axes that barely matter. Random search covers the important axes better at the same budget, and Bayesian optimisation spends the budget where the surface is promising. Set the budget first, and never let the search see the test set.Grid Search and Random Search → |
| "AutoML picked the model, so it's tuned" | AutoML ran a search you did not see, against a validation split it chose, under a budget it set — and the reported number is the best of many attempts, which is optimistic by construction. Ask what it searched, on what split, and evaluate the winner yourself on a period it never touched.AutoML Hides the Search → |
| "The forecast is great for tomorrow and useless for next month" | Error grows with the horizon, and a model validated only at one step ahead says nothing about thirty. Evaluate at every horizon the decision uses, with rolling-origin validation so each forecast is made from data that existed at its origin, and be explicit about where the forecast stops being better than a seasonal naive.The Forecast Horizon → |
| "The recommender keeps showing the same ten items" | A feedback loop: the model recommends what got clicks, clicks happen on what was recommended, and next week's training data contains only those items. Without exploration the catalogue narrows toward what was already popular, and the offline metric on logged data will say it is doing well. Reserve traffic for exploration and log the propensity.Feedback Loops → |
| "We can't reproduce last month's model" | Reproducibility needs the data version, the feature code, the training code, the config, the seed and the environment — a seed alone reproduces nothing if the table changed. Track every run with all six, version the datasets, and keep the lineage from artifact back to inputs so the question "what was this trained on" has an answer.Reproducibility → |
| "Two runs with the same seed gave different results" | Seeds control the pseudo-random streams you own; they do not control non-deterministic GPU kernels, thread scheduling, data-loader ordering across workers or a library version change. Decide how much determinism you need, pay for it where you need it, and treat run-to-run variance as a measurement — it is the noise floor of your metric.Random Seeds → |
| "The model file is in a shared drive somewhere" | An artifact without a registry has no version, no lineage, no stage and no way to say which one is in production. A registry records candidate → staging → production → archived, attaches the evaluation and the lineage, and makes rollback a lookup rather than an archaeology project.The Model Registry → |
| "The scaler is in the notebook, the model is in the artifact" | The model expects inputs shaped by preprocessing that lives somewhere else, so any serving path that reimplements the scaler is a skew waiting to happen. Put the fitted preprocessing inside the artifact so the thing you deploy accepts raw features and does the same transformation as training, by construction.Preprocessing Lives in the Artifact → |
| "We need real-time predictions" (for a nightly email) | Freshness is decided by when the decision is made, not by how the request arrives. If the prediction is consumed once a day, batch-scoring every entity overnight is cheaper, simpler and easier to monitor than an online service, and it makes the feature computation identical to training. Online inference is for decisions that need features that did not exist a minute ago.Choosing the Inference Mode → |
| "The GPU is at 15% utilisation" | Requests are arriving one at a time and the GPU spends most of its life waiting for the next one; the cost per prediction is set by that idle time. Dynamic batching groups requests within a small window, raising utilisation at the price of a few milliseconds of latency — and if the batches stay small, a CPU may be the cheaper answer.Inference Batching → |
| "The feature store project has taken six months" | A feature store solves train/serve skew and point-in-time correctness for teams that share features across many models with online and offline paths. For one model on a batch pipeline, the same guarantees come from computing features once and writing them to a table. It is infrastructure with a cost; whether it is worth it is a question about your team, not a rule.Feature Stores → |
| "The feature is fresh in training but hours stale in production" | Training used the value as of the event; production reads whatever the last pipeline run wrote. The model learnt from information it will not have at serving time — a freshness skew. Either serve the feature at the freshness training assumed, or train on the feature at the staleness production will have, and make the choice explicit.Feature Freshness → |
| "We'll run it in FP32 because accuracy matters" | Most models lose nothing measurable at INT8 or FP16 for inference, and the win is not only speed: a quantized model needs less memory bandwidth, which is what inference on an accelerator is usually bound by. Measure the quality on your validation set at each precision instead of assuming, and keep the FP32 model as the reference.Quantization → |
| "The training job died at hour 40" | A long run without checkpoints is a bet that nothing fails for its whole duration, and at scale something always does. Save model, optimizer and step state at intervals whose cost is small next to the cost of losing the run, and test the resume path — a checkpoint you have never restored from is not a checkpoint.Checkpointing → |
| "All the unit tests pass, so the model is fine" | Unit tests cover code; a model fails through data, features, training and drift, none of which a green suite touches. The ML test stack adds data and schema tests, a training smoke test on a small sample, invariance tests, a serving contract test and a regression suite against the current champion — each catching a class of failure the others cannot.The ML Testing Stack → |
| "Changing the customer's name changes the fraud score" | An invariance test would have caught it: for inputs that should not affect the prediction, perturb them and assert the output does not move. The model found a spurious signal in a field it should be indifferent to. Encode such invariants as tests so a retrain cannot silently reintroduce them.Model Invariant Tests → |
| "The model is accurate but the approval rate differs by group" | Report subgroup performance — error rates, calibration, positive rates — per protected group, and expect that not every fairness definition can be satisfied at once; equal accuracy and equal positive rates conflict when base rates differ. Which one matters is a product and legal decision that the model cannot make.Fairness → |
| "The SHAP plot explains the prediction" | It approximates the model's local behaviour with a simpler function; it explains the model, not the world, and different methods give different explanations of the same prediction. Use it to debug and to sanity-check, present it as approximate, and never let an explanation be the reason a decision is defended if the model itself would not be.Explainability → |
| "The prediction distribution shifted but nothing else changed" | Prediction drift with stable features is a feature pipeline change, a model version change or a bug in serving before it is anything about the world. Check the model version and the feature values on a sample of requests. If those are stable, the model is responding to a feature shift you are not monitoring.Prediction Drift → |
| "Drift alert fired, so we kicked off retraining" | Drift is a change in a distribution; it says nothing about whether quality fell. A new marketing campaign shifts the features and the model may be fine; a feature bug shifts them and retraining bakes the bug in. Measure quality on the drifted slice, find the cause, and only then decide whether new training data is the fix.Drift Is Not Failure → |
| "Quality has been sliding for three months" | Gradual decay is the signature of the world moving away from the training window — but confirm it: plot quality per cohort against the age of the model, rule out a slow feature-pipeline degradation, and check whether the model's own decisions changed the population. Then retraining on recent labels is a reasonable answer, with the old model as the challenger.Performance Decay → |
| "The challenger beat the champion offline" | Offline is one dataset and one split. Run the challenger in shadow against the champion on live traffic, compare on the same requests, then A/B on the business metric with a sample size chosen in advance. The champion has a track record in production that the challenger has to earn, not inherit from a validation number.Champion / Challenger → |
| "The A/B test was significant after two days" | Peeking at a test and stopping when it crosses a line inflates the false-positive rate; two days also misses weekly seasonality and novelty effects. Fix the sample size and duration before starting, define the metric and the guardrails in advance, and make sure the randomisation unit matches the unit of the outcome.A/B Testing Models → |
| "We can't tell which model version made that prediction" | Prediction logging that does not record model version, feature values and request id cannot be joined to outcomes, cannot be traced and cannot be replayed. Log enough to reconstruct the prediction — version, inputs as seen, output, decision — under a retention and privacy policy, so an incident can be walked back from the outcome.Prediction Logging → |
| "We need Kubernetes to do MLOps" | MLOps is a set of practices — versioning, testing, automated training, deployment with rollback, monitoring — and each can be met with a cron job and a registry as well as with an orchestrator. Choose the platform from the number of models, teams and the release cadence, not from the word. A single model retrained weekly does not need a cluster.What MLOps Is → |
| "CI passed, so the retrained model can deploy" | CI tested the code. Continuous training tests that a training run produces a model that meets the bar on fresh data, and continuous delivery tests that the artifact serves correctly under the contract. The three are different gates; a pipeline that conflates them deploys a model that trained successfully on broken data.CI, CT and CD for ML → |
| "The cloud bill for training tripled" | Look at idle accelerators, oversized instances, runs nobody looked at, and re-computation of features that could be cached — before looking at model size. Most ML cost is waste around the run rather than the run itself, and a spot instance with checkpointing often halves the rest.ML Cost Optimisation → |
| "Anyone can submit training data through the form" | Data poisoning: an attacker who can influence the training set can move the model — a spam classifier taught that their template is legitimate, a recommender taught to surface their item. Validate, rate-limit and audit the sources of training data, and treat user-generated labels as untrusted input.Data Poisoning → |
| "We pulled the pretrained weights from a public repository" | A model artifact is executable content with a supply chain: the weights, the loader, the tokenizer and the dependencies. Pin versions, verify hashes, load in a format that does not execute arbitrary code, and record the provenance in the registry — the same discipline as any third-party binary.The Model Supply Chain → |
| "Someone is calling the scoring API a million times a day" | Inference abuse: an endpoint that returns scores can be used to extract the model, probe its decision boundary or find inputs that evade it — and it costs you compute either way. Authenticate, rate-limit, return decisions rather than raw scores where possible, and monitor for query patterns that look like a search.Inference Abuse → |
| "Design a fraud detection system" | Ask before drawing: what is the decision and its latency budget, what does a false decline cost against a missed fraud, how late do labels arrive, how imbalanced is it, and how will fraudsters adapt? Then features with point-in-time correctness, an online model with a fallback, threshold by cost, delayed-label monitoring and an adversarial retraining loop.Designing a Fraud Detection System → |
| "Should this be an ML model or an agent?" | If the task is a prediction from structured inputs with a measurable outcome and a latency budget, it is a model. If it needs to read, reason, call tools and produce an open-ended output, it is an agentic system — built on models, evaluated differently, and taught in Agentic Engineering. Most products need both, and the boundary is where the prediction becomes an action.The Boundary With Agentic Engineering → |