Machine learning interview guide
Every question carries the same frame: the production scenario behind it, what it is actually testing, what a strong answer sounds like, and the flags on both sides. The red flags are specific wrong answers a confident engineer really gives — an algorithm named before the data was asked about, a number trusted before the split was.
A fraud team has spent three months on a transaction classifier that hovered around a validation AUC in the low 0.8s (illustrative). A new engineer joins, adds a handful of features from the operational warehouse, and the next morning validation accuracy is 99.9% and AUC is 0.999. The engineer has opened a pull request titled "new features — huge win" and the product manager wants it in production this week.
A payments company is evaluating a vendor fraud model. The vendor deck reports 99.85% accuracy on a held-out set. The internal team has a rule-based system that blocks transactions from a list of known bad merchants and card BINs, and nobody has ever computed its accuracy. Leadership wants to know whether the vendor model is better than what exists.
A subscription business has a churn model that outputs a probability per customer each week. The retention team calls every customer above 0.5 with a discount offer. They have a budget for 2,000 calls a week, the model flags 9,000 customers, and about one in five of the customers called would have churned anyway (illustrative). The data scientist's answer to "why 0.5" was "that is the default".
A demand forecasting model for a grocery retailer was validated on the last twelve weeks of history with a healthy error reduction against the incumbent (illustrative). Three weeks after deployment, store managers report over-ordering of fresh produce and the waste metric has gone up. The offline evaluation, re-run today on the same validation set, still shows the same strong numbers.
A credit-risk team has a drift monitor on every input feature. On Monday morning it fires on `device_type`: the share of requests from a new mobile app version jumped from near zero to 40% over the weekend after a release (illustrative). The on-call engineer's runbook says "drift detected: trigger retraining". Decisions from this model take 60 to 90 days to resolve into a label.
A product manager for a marketplace asks the ML team to "build a model that predicts which listings will sell in the first week". The data is roughly 200,000 listings a month with title, description, photos, price, category, seller history and a sold-within-7-days label. The prediction feeds a seller-facing "improve your listing" hint and, later, maybe ranking. The PM asks in the kickoff: "so which model — XGBoost?"
A streaming subscription service has monthly plans, annual plans, free trials, involuntary churn from failed payments, and customers who pause for a season and come back. A director has asked for "a churn model". The data science team has three different definitions in three notebooks, and the numbers they report are not comparable.
A logistics company is building a model to predict whether a delivery will be late. The training table was built by joining shipments to scan events, and it has 40 million rows. Some shipments have one row, some have thirty. The team reports AUC on a random split and is pleased. The model is meant to run once, when the shipment is created.
An energy company forecasts hourly load per region 24 hours ahead. Three years of hourly data exist. The team's first model used five-fold cross-validation with shuffled folds and reported an error far better than the incumbent statistical forecast; the incumbent team says the comparison is unfair.
A healthcare startup classifies skin lesion images. Each patient contributed several photos of the same lesions over time. The random split by image gives a strong validation result; a pilot at a new clinic, with new patients, performs much worse. The team suspects the clinic's cameras.
An ad-tech team has a click model with a `site_id` feature of 300,000 distinct values. One-hot is impossible, so an engineer replaced it with the mean click rate per site, computed over the full training table, and validation log loss improved sharply (illustrative). The engineer describes this as "the model learned which sites are good".
A support team wants a model to predict ticket resolution time so they can set customer expectations. A contractor delivered a gradient-boosted model with a validation MAE of 6.1 hours (illustrative) and a slide saying "state of the art". Nobody knows what the MAE of "the median for this ticket category" would be.
An insurer uses a claim-likelihood model to price quotes: expected cost equals probability of claim times expected claim size. The model has a high ROC AUC (illustrative). After launch, the book of business is unprofitable in the low-risk segment and quotes are uncompetitive in the high-risk one. The model was trained with heavy downsampling of non-claims.
A retailer forecasts daily units sold per product per store. Most product-days sell zero to five units; a few sell hundreds. Finance wants a single accuracy number in a percentage. The current dashboard reports MAPE, which is dominated by products that sold one unit and were forecast at two, and is undefined on the days that sold zero, which are silently excluded.
A team training a document classifier is deciding whether to buy more labelled data at a significant cost per label. They have a plot of training and validation loss against training-set size, and another against epochs. The tech lead's proposal is "more data always helps".
A pricing team runs a gradient-boosted model with 3,000 trees that was tuned last year. A colleague proposes "adding another 2,000 trees to improve it" and separately proposes averaging it with a random forest "because ensembles are always better". Both suggestions came from the same intuition that more trees are better.
A team is fine-tuning a small transformer for intent classification. Training looks fine for a few hundred steps, then loss spikes and becomes NaN. The engineer's first fix was to restart with a different seed, which worked once and then failed again at a different step.
A retailer built a "similar products" feature from product embeddings and cosine similarity. It works well for shoes and badly for groceries, where a bag of rice is "similar" to a bag of flour and also to a bag of charcoal. A 2D plot of the embeddings in a slide deck looks convincing.
A customer-support platform classifies incoming tickets into 40 routing categories. A prompted hosted LLM gets acceptable accuracy at a cost per ticket that is too high for the volume, and its outputs occasionally drift when the provider updates the model. The team has 300,000 labelled historical tickets.
A team ran 1,200 configurations of a gradient-boosted model through random search, selected the best on the validation set, and reported its validation AUC as the expected production AUC. Six months later the model underperforms that number by a margin no drift monitor explains.
A news app has a homepage recommender trained on clicks. New articles get few impressions and few clicks, so they rarely surface; new users see the globally popular feed. A product review found the homepage increasingly narrow: the same dozen topics, and a click-through rate that rose for two months and then fell.
A regulator asks a lender to explain a decision made eight months ago by a credit model. The team has the model file, the git commit of the training code and a fixed random seed. Retraining from that commit produces a model with different coefficients and a different decision for the case in question. Nobody can say which is the one that ran.
A bank scores every customer nightly for a pre-approved credit-line offer and shows the result in the app the next day. Product wants the offer to reflect a deposit made a minute ago, so they ask to "make the model real-time". The training set was built from a nightly warehouse table.
A team serves a gradient-boosted risk model and a small transformer for text classification from the same CPU fleet. Latency complaints are about the transformer at p99. An infrastructure proposal moves both to GPU instances at roughly triple the cost and separately proposes INT8 quantization "for free speed".
A team trains a ranking model on one GPU in 26 hours. They are given eight GPUs and expect training in about three hours. Their first attempt takes nine hours and produces a slightly worse model. Separately, a larger model they want to try does not fit in one GPU's memory at all.
A team ships a retrained model through CI with full test coverage on the training and serving code. Two weeks later they discover that a schema change upstream renamed a column, the feature pipeline silently filled it with zeros, the model trained on zeros, and the serving path served zeros. Every test passed and the offline metric dropped only slightly, within the range attributed to noise.
A hiring platform uses a model to rank applicants for recruiter review. Compliance asks whether it is fair. The team removed gender and ethnicity as features and reports that the model "cannot discriminate because it does not see those attributes". Selection rates differ by group; so do the base rates in the historical hiring data the model was trained on.
A lending model's dashboard shows a rolling precision of 100% for the last four weeks and the team has stopped looking at it. Defaults are only known 30 days after the due date, and the dashboard counts every prediction with no label yet as "correct so far". A new model was deployed three weeks ago.
A search team has a new ranking model that is better on offline metrics (illustrative). The prior model is in production serving all traffic. The new model uses two new features from a service that has never taken production load. Business asks for it to be live by the end of the quarter.
A health-insurance recommender logs the full request payload, including free-text notes, to a shared log cluster with 90-day retention so that "we can debug anything". Privacy has flagged it. Meanwhile, the on-call engineer cannot answer "why did member X get this recommendation on Tuesday" because the log has the input but not the model version, the feature vector or the threshold.
A marketplace trains its spam-listing classifier on listings that users report, plus the outcome of moderator review. A group of sellers has discovered that mass-reporting a competitor's listings, and creating many innocuous listings with a distinctive phrase and then reporting their own, shifts the model. The retraining job runs nightly and promotes automatically if the offline metric holds.
An operations team wants to "use AI" to triage incoming supplier invoices: extract fields, flag likely duplicates and anomalies, and route exceptions to a human. Volume is 40,000 invoices a month. One proposal is an agent with tools that reads each invoice; another is a set of trained models. A third proposal is "both, somehow".