Positional Information
Attention is a weighted sum over a set: shuffle the tokens and it computes the same thing. Order has to be injected explicitly — learned, sinusoidal, relative or rotary — and the scheme you pick decides whether the model can say anything sensible past the lengths it was trained on.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Attention treats its input as a set. How does a transformer know which token came first, and why does a model degrade on inputs longer than it was trained on?
A team built a log-anomaly detector on a transformer over event sequences. It was trained on windows of 512 events and works. Operations now wants to run it over 4,000-event windows to catch slow-building incidents, and the first attempt returns confident nonsense on the longer windows even though every individual event is one the model has seen thousands of times.
The transformer reads sequences; give it a longer sequence. The attention will just have more positions to attend over, and every event is familiar.
Attention itself has no idea what position anything is at. It computes a weighted sum over the set of token vectors; permute the input and every score, every softmax and every output permutes identically. Order reaches the model only through whatever was added to the embeddings, and the model has never seen what "position 3,000" looks like.
- Attention itself has no idea what position anything is at. It computes a weighted sum over the set of token vectors; permute the input and every score, every softmax and every output permutes identically. Order reaches the model only through whatever was added to the embeddings, and the model has never seen what "position 3,000" looks like.
- With learned absolute positions the embedding table has 512 rows and no row 3,000 — it is literally undefined. With a fixed scheme the row exists but the attention weights were tuned for scores from positions under 512, and the unfamiliar scores push the softmax into regions the model never trained in. Either way "confident nonsense" is the honest description.
- Offline evaluation on 512-event windows showed nothing, because it could not. The failure exists only at lengths the validation set, cut with the same limit, did not contain.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a window of service events contains an anomaly; the label is an incident ticket opened within the window's span, and the decision is paging an on-call engineer.
- Order carries the signal: "restart, then error" is a recovery and "error, then restart" is a fault, so a model that cannot represent order cannot learn the target.
- One example is a sequence of event tokens — service, level, message template — with a binary label from tickets. Training windows were cut at 512 events because that was the model's configured length.
- The sequences are inherently ordered and the positions mean "how many events ago", which is a relative notion; nothing about position 300 is special except its distance from position 301.
- The 4,000-event windows contain positions the model has never occupied, in a distribution of event types that is otherwise unchanged.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Self-attention is permutation-equivariant: applying a permutation to the rows of X permutes the rows of the output identically, with no other change. So a transformer without positional information is a set model, and a classifier pooled over the outputs is fully permutation-invariant — "error then restart" and "restart then error" are the same input.
- Absolute schemes add a vector per position to the token embedding before the first block. *Learned* absolute positions are a table with one row per position up to a maximum; nothing exists past it. *Sinusoidal* positions use fixed sine and cosine functions at a range of frequencies, so any integer position has an encoding and nearby positions have similar ones — but the model still only saw the first 512 rows during training.
- Relative schemes encode the *offset* between query and key positions rather than the absolute index — a bias on the score that depends on j−i, or, in rotary embeddings, a rotation of q and k by an angle proportional to position so the dot product q_i·k_j depends only on j−i. Relative schemes generalise better to longer inputs because "three events ago" means the same thing at position 3,000 as at position 30; they still degrade when offsets exceed the trained range, since the attention has never seen a score from a distance of 3,500.
Attention is a set operation
Write out what attention does to a permuted input. Reorder the rows of X; then Q, K and V are reordered identically, the score matrix has its rows and columns permuted together, the softmax acts row-wise so it is unaffected, and the output rows come out in the permuted order carrying the same values. Nothing in the computation referenced an index.
The feed-forward layer is applied per row and cannot mix positions at all. So a transformer with no positional signal is a function on multisets of vectors, and "error, then restart" is indistinguishable from "restart, then error". Order enters only through what you add to the embeddings, and that addition is the entire mechanism this lesson is about.
1import numpy as np2rng = np.random.default_rng(0)3n, d = 4, 84X = rng.normal(size=(n, d))5Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))6 7def attn(X):8 S = (X @ Wq) @ (X @ Wk).T / np.sqrt(d)9 W = np.exp(S - S.max(1, keepdims=True)); W /= W.sum(1, keepdims=True)10 return W @ (X @ Wv)11 12perm = [2, 0, 3, 1] # "restart, error" -> "error, restart"13assert np.allclose(attn(X)[perm], attn(X[perm])) # same rows, reordered14# a classifier that pools over rows therefore sees the SAME input either wayThe assertion passes for any weights. That is the point: there is no setting of W_Q, W_K, W_V under which attention alone can distinguish orderings, so no amount of training data fixes it. Position has to be an input.
Four ways to say where a token is
Absolute schemes tag each position with a vector added to its embedding. A learned table is the simplest and has a fixed number of rows. Sinusoids give every integer a distinct encoding whose components vary at different frequencies, so nearby positions are nearby vectors; the model learns to use that geometry, but only over the range it trains on.
Relative schemes put position into the score rather than the embedding: a learned bias per offset j−i, or a rotation of q and k by position-dependent angles so their dot product depends only on the offset. Because the anomaly detector's signal is "how many events ago", relative encodings match the target's structure — and they still see novel offsets when the window grows eightfold.
Recall of ticketed incidents on held-out 512-event windows was strong and stable across three months of validation sets.
On 4,000-event windows the model paged on-call several times a night with high confidence and missed two real incidents in the first week.
- 1Positions past 512 are outside the trained range — undefined for a learned table, unfamiliar for any scheme — so the attention scores land in regions the softmax was never calibrated in.
- 2The validation sets were cut at the same 512-event limit and so could not contain the failure; the offline number was correct about a system that was not deployed.
- 3Long windows also dilute the softmax mass over the few anomalous events, which lowers their contribution even when the positions are handled.
| Scheme | Where position enters | Beyond trained length | Cost |
|---|---|---|---|
| Learned absolute | A table row added to each embedding | Undefined — no row exists | Negligible; one vector per position |
| Sinusoidal absolute | Fixed sin/cos vector added to each embedding | Defined but unfamiliar; weights never saw those inputs | Negligible; no parameters |
| Relative bias | A learned bias on score(i, j) by offset j−i | Offsets past the trained range are clipped or unseen | Small; a bias table per head |
| Rotary | q and k rotated by position; dot product depends on offset | Degrades gradually; extendable with base scaling and a short fine-tune | Small per-score rotation; no extra parameters |
What must stay true about length
The model encodes a promise about the positions it will be asked about. Extending the served length without extending the training data breaks it in the most invisible way this domain has: no exception, familiar tokens, confident scores, and an evaluation set that structurally cannot contain the failure.
So length is an assumption to state, guard and monitor. State the validated range in the model card; guard it at the serving boundary with a fallback that is a decision rather than an accident; monitor the served length distribution so the next request to extend the window arrives as a planned fine-tune rather than a night of false pages.
Every window scored in production has a length — and therefore a set of absolute positions or relative offsets — within the range the model was trained and validated on, or is mapped into it by a validated policy.
holds when The serving boundary enforces a maximum length, training windows were sampled at the served lengths, and the recall-versus-length curve is flat across the served range.
breaks when A consumer extends the window; a collector change produces denser logs so the same time span has more events; an interpolation policy is applied without re-validation.
respond Reject or chunk out-of-range windows at the boundary today; schedule a length-extended fine-tune with a validation set at the new length; do not raise the configured limit and hope.
How to build it
Most important first.
- Choose the positional scheme for the extrapolation you will need, not the length you have. Relative or rotary schemes are the default for anything that may run longer than trained; learned absolute positions are a hard ceiling.
- Train on the lengths you serve. If 4,000-event windows are the product, the training and validation windows must include them; the cheapest fix for the anomaly detector is a length-extended fine-tune (Fine-Tuning).
- When the model must run past its trained length, use a policy rather than hoping: slide a 512-event window and aggregate scores, or interpolate the positional scale so 4,000 positions map into the trained range — and validate the policy on long windows with known incidents.
- Put a hard input-length guard at serving time with an explicit fallback, so a longer window is rejected or chunked deliberately rather than scored on undefined positions (Serving Fallbacks).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Recall of ticketed incidents at the paging threshold on held-out windows *of the production length*, bucketed by length; this is the number that maps to the decision and it is the one the 512-cut validation set could not produce.
- A permutation test on a sample: shuffle the events within a window and check that the score changes. A model whose score survives shuffling has learned event frequencies, not order.
- Do not measure "accuracy per event type". Every event type is familiar; the failure is positional and shows only as a function of length.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The lengths served in production fall within the range of positions — absolute or relative — the model was trained on, or a validated policy maps them into it.
- Order carries the signal the model learned; a data change that reorders events — a log collector that batches by service rather than by time — destroys the feature the positional scheme exists to represent (Data & Feature Tests).
- The positional scheme in the serving artifact is the one the weights were trained with; a re-implementation that computes rotary angles with a different base is silent skew.
- Offline: evaluate on windows at 1×, 2×, 4× and 8× the trained length with known incidents, and plot recall against length; the curve is the extrapolation behaviour and it should be flat where you serve.
- Online: log the window length per request and alert when the p99 exceeds the validated range; keep a shadow of the sliding-window policy running on long windows and compare paging decisions (Shadow Deployment).
- Over time: re-run the permutation test on a fresh sample monthly; a score that becomes insensitive to shuffling means the collector's ordering, or the model's dependence on it, has changed.
What can go wrong
- Operations extends the window again, to 16,000 events, and the length-extended model degrades the same way, one length later; the assumption was re-broken, not fixed.
- The sliding-window policy aggregates by max, and a single spurious high score in one chunk pages on-call for every long window.
- Rotary interpolation squeezes 4,000 positions into the trained range and now adjacent events look too close together for the model to distinguish "immediately after" from "a few events later" — the recovery/fault distinction the target depends on.
- Relative schemes cost a little extra computation per attention score and generalise better; learned absolute positions are cheapest and simplest and stop dead at the table's end.
- Training on long windows multiplies attention cost quadratically; a sliding-window policy keeps cost linear and loses cross-window context, which for slow-building incidents is the context you wanted.
- A hard length guard prevents undefined behaviour and turns the longest windows into rejected requests; whether that is acceptable is a product decision about what "we could not score this" should trigger.
- "Every event in the long window is one the model has seen, so it should work." The events are familiar; the positions are not. A transformer does not see "an event"; it sees an event vector plus a positional signal, and half of that input is out of distribution.
- "Sinusoidal positions are defined for any length, so the model can extrapolate." The encoding is defined; the attention weights that consume it were trained on a bounded range. Definedness is not generalisation.
- "The model can just learn order from the data." Without positional information there is nothing to learn it from: the architecture computes the same output for every permutation, so no loss can distinguish orderings.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- SIMPLIFIEDLearned, sinusoidal, relative and rotary schemes are described at concept level; the exact rotary construction, ALiBi-style score biases, and the interpolation and extension tricks used to stretch a trained context are named but not derived, because the extrapolation argument does not depend on their details.
- GENERALPermutation equivariance and the need for explicit positional information hold for every attention-based model on ordered data, from event logs to language to time series. What is task-specific is whether order carries the signal — for bag-of-words sentiment it barely does, for log sequences it is the whole target.
- CONTESTEDA serious position holds that length extrapolation is largely solved in practice — that rotary embeddings with scaled bases or ALiBi-style biases, plus a short length-extension fine-tune, give models that work well past their trained length, and that the "hard ceiling" framing is dated. The practical record supports modest extension with those techniques; the disagreement is about how far, and every extension reported still degrades somewhere, which is why the lesson insists on validating at the served length rather than trusting the scheme.
Where the depth lives
This domain teaches the model and hands the rest off by name.