Adversarial Inputs
Small, deliberate changes to a valid input flip the prediction. Defensively: validate content, test robustness, use ensembles and monitor confidence — and accept that fraud and spam are adversarial by nature.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A valid-looking input produces a confidently wrong prediction, and slightly different versions of it produce the right one. What property of the model does that expose, and what defends against it without knowing the attack?
Our spam filter's catch rate has been sliding for a month while its validation metrics on the labelled set are unchanged. The messages getting through are readable and obviously spam to a person, but they carry odd spacing, look-alike characters and image text. Retraining on last month's data helped for a week. The people sending them are clearly testing what gets through.
A classifier that scores well on held-out data has learned what spam looks like. If new spam gets through, add it to the training set and retrain. The metric will recover.
The held-out set was drawn from the same distribution as training, before anyone adapted to the model. It measures performance against last month's spam. The senders are producing this month's, by probing the model directly.
- The held-out set was drawn from the same distribution as training, before anyone adapted to the model. It measures performance against last month's spam. The senders are producing this month's, by probing the model directly.
- Retraining on the new variants teaches the model those variants; the senders produce the next ones within days. The retraining cadence is now set by the adversary (Retraining as a Decision).
- The model's confidence on the evasive messages is high in the wrong direction, so a confidence threshold does not catch them; they were constructed to sit confidently on the wrong side (Calibration does not survive adversarial inputs).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Classify a message as spam or not; the label comes from user reports and reviewer decisions. The defensive target is a classifier whose decision boundary does not move far under small, semantics-preserving changes to the input, and a monitor that notices when inputs start clustering near that boundary.
- Messages with text, sender features and metadata. The training set is historical spam and ham; the validation set is a random split of it. Neither contains the look-alike-character variants, because those were produced after the model was trained, by people reacting to it.
- A decision tree over token features is part of the ensemble; its splits are on the presence of specific tokens, which is why a character substitution defeats it.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A model partitions input space with a decision boundary. Near that boundary, a small change in the input changes the output. High-dimensional inputs — text, images — have many directions in which a small change is semantically invisible to a person and large to the model, because the model weighted features a person does not attend to: specific tokens, pixel patterns, spacing. An adversarial input is a valid input moved across the boundary along one of those directions.
- The property this exposes is sensitivity: how much the prediction moves for a small perturbation. It is a property of the learned function, not of any particular attack, which is why the defence can be stated without one. Robustness testing measures it directly (Robustness Testing); model-invariant tests assert that specific transformations should not change the output (Model Invariant Tests).
- Spam, fraud and abuse detection are adversarial by nature: the inputs are produced by people who observe the model's decisions and adapt (Designing a Fraud Detection System). For these, robustness is a design requirement, not a hardening step, and the training distribution is by definition stale.
The boundary is closer than it looks
The tree visualizer at /ml/tree grows a decision tree on two-dimensional points one level at a time. Watch where the splits fall: a deep tree draws boundaries that pass close to many training points, and a point nudged slightly across a split changes class. In two dimensions that nudge is visible. In the thousands of dimensions of a token vector, there is almost always some direction in which a small nudge crosses a split and a person sees nothing.
That is the mechanism, stated without any attack. A model's sensitivity is measurable on benign data: apply a transformation that should not change the answer, and count how often it does.
Add last week's look-alike-character messages to the training set and retrain. Catch rate recovers for a week; the senders change the substitution; repeat. The team's cadence is now set by the senders.
Normalise characters and spacing before the model; extract image text; add a character-level view to the ensemble; gate deployment on a robustness test over semantics-preserving transformations; monitor inputs near the boundary.
Retraining moves the boundary to cover the last observed inputs; reducing sensitivity makes the boundary harder to cross along the directions a person does not see. The second raises the adversary's cost per evasion; the first raises yours per retrain.
The metric was measured before anyone adapted
The validation set is a sample of the world before the model existed in it. In a non-adversarial domain that is a manageable staleness. In an adversarial one it is the wrong distribution by construction: the inputs that matter are the ones produced in response to the model, and they cannot be in a set drawn before the model shipped.
The offline/online device is the sliding catch rate against the unchanged validation metric, with the mechanisms in the order they most likely explain the gap.
Validation metrics on the labelled historical set unchanged; robustness never measured.
Catch rate from user reports sliding week over week; the messages getting through are evasive variants that did not exist when the validation set was drawn.
- 1The senders observe which messages are delivered and adapt: character substitutions, spacing and image text move messages across the token model's boundary.
- 2The ensemble's views share token features, so an evasion of one is an evasion of most.
- 3The validation set predates adaptation and cannot contain the variants; the metric is stable because the distribution it measures is stable.
A robustness test that does not need to know the attack
The test applies transformations that should not change the answer and counts how often they do. It needs no knowledge of what the senders will try next; it measures a property of the model. The transformation set grows as evasions are observed, but its purpose is to bound sensitivity in general, not to replay last week.
The assumption device states what must remain true for the whole defence — normalisation, ensemble, monitoring — to keep working, and how the team would know it stopped.
Semantics-preserving transformations of an input leave the model's decision unchanged, so that the boundary cannot be crossed along directions a person does not perceive.
holds when Inputs are normalised before scoring; the ensemble's views are independent; the robustness gate runs on every candidate and its transformation set is maintained.
breaks when A new model with a single feature view replaces the ensemble for latency; the normaliser is bypassed by a new input path such as image text; the transformation set stops being extended after observed evasions.
respond Fix the sensitivity — normalise, add a view, retrain with the transformation as augmentation — before retraining on the evasions alone; and route boundary inputs to review while the fix lands.
1def flip_rate(model, messages, transforms, threshold):2 # transforms: functions that should NOT change the meaning of a message3 # e.g. unicode-normalise look-alikes, collapse spacing, swap a synonym4 flips = {t.__name__: 0 for t in transforms}5 for msg in messages:6 base = model.score(msg) >= threshold7 for t in transforms:8 if (model.score(t(msg)) >= threshold) != base:9 flips[t.__name__] += 110 rates = {name: n / len(messages) for name, n in flips.items()}11 # a transformation that flips more than a small fraction is a sensitivity to fix,12 # not a reason to add the transformed messages to training13 return {name: r for name, r in rates.items() if r > 0.02}The transformations are benign by construction — a person would say the meaning is unchanged. That is what makes the flip rate a property of the model rather than a replay of an attack.
How to build it
Most important first.
- Validate content, not just schema: normalise look-alike characters and spacing before the model sees the input, extract text from images, so that the semantics-preserving variants collapse to one representation (Parse, Validate, Authorize, Process in Security Engineering is the general discipline).
- Test robustness in the pipeline: apply a set of semantics-preserving transformations to the validation set — character substitution, spacing, paraphrase — and gate on the prediction not changing. This measures sensitivity without knowing the next attack.
- Use an ensemble of models with different feature views — tokens, character n-grams, sender behaviour, image content — so that evading one view does not evade the decision (Random Forests for why diverse views help).
- Monitor the distribution of prediction confidence and of inputs near the boundary; a rise in inputs clustering just below the threshold is a probe in progress. Route those to review rather than auto-approve.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Robustness score: the fraction of validation predictions that survive the transformation set unchanged. This is the number that says whether the boundary is sensitive along known-benign directions.
- Catch rate on live traffic from user reports, with a short delay, per week — the only number that reflects this month's spam.
- Validation metrics on the historical set are the numbers that look relevant and are not: they measure the distribution before the adversary adapted.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The transformations in the robustness test represent the semantics-preserving changes the model should ignore, and the set is extended whenever a new evasion is observed.
- The ensemble's feature views are genuinely independent — evading the token model does not evade the sender-behaviour model — and this is checked by measuring their agreement on evasive inputs.
- Inputs that land near the decision boundary are routed to review, and the review outcomes feed back as labels through a process the senders cannot influence (Data Poisoning).
- Offline: run the transformation set on the validation data and measure the flip rate per transformation. A transformation with a high flip rate is a sensitivity to fix before deployment.
- Online: track the confidence distribution and the share of inputs within a narrow band of the threshold per day; a rising share is a probe, and the messages in it are the ones to inspect.
- Over time: compare catch rate from user reports against the validation metric each month. A widening gap is adaptation, and the response is a robustness fix rather than a plain retrain.
What can go wrong
- Normalisation collapses look-alike characters, and the senders move to a substitution the normaliser does not cover; the normaliser becomes its own arms race.
- The robustness gate tests transformations that were chosen from last quarter's evasions and does not include the one that appears next.
- The ensemble's views are diverse in principle and correlated in practice, because three of four are trained on the same token features.
- Input normalisation and image-text extraction add latency and their own maintenance, and each normaliser is a rule that the next evasion targets.
- Ensembles cost inference — several models per message — and complicate explanation and rollback.
- Routing boundary inputs to review is a queue that grows exactly when a probe is in progress.
- "Validation is stable, so the model is fine and the reports are noise." Validation is on the distribution before adaptation. The reports are the current distribution.
- "Retrain more often." Retraining on the adversary's last move sets your cadence to theirs. Reduce sensitivity — normalise, diversify views — so the next move costs them more than it costs you.
- "Use the confidence to filter." Adversarial inputs are constructed to be confidently wrong. The confidence distribution is a monitor for probes, not a filter for individual inputs.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a decision boundary in a high-dimensional input space is sensitive along directions a person does not perceive is a property of learned models generally; how sensitive, and along which directions, varies by model family and input modality.
- DOMAIN-SPECIFICIn spam, fraud and content moderation the inputs are produced by parties who observe and adapt to the model, so adversarial inputs are the normal condition; in demand forecasting or churn nobody is crafting inputs, and the same sensitivity shows up only as fragility to benign noise.
- MODEL-SPECIFICA tree over token presence flips on a single character substitution; a character-level or embedding-based model is less sensitive to that and sensitive to other perturbations. Diverse model families in an ensemble help precisely because their sensitive directions differ.
Where the depth lives
This domain teaches the model and hands the rest off by name.