Sequence Models
A recurrent network carries a hidden state step by step through a sequence; that is elegant and it is why long dependencies were hard. Transformers replaced the recurrence with attention so every position can be computed in parallel — and simpler models still win many forecasting problems.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Recurrent networks read a sequence one step at a time. Why did that make long-range dependencies hard to learn, and what did transformers change about the computation?
A support team wants to route incoming tickets by reading the first message. An older model — a recurrent network over the tokens — routes short tickets well and misroutes long ones, where the actual request is buried after three paragraphs of context. Someone has proposed "just use a transformer"; someone else has asked why the team's demand forecast, which also handles sequences, should not switch too.
A sequence is a sequence. Feed tokens one by one into a recurrent cell that updates a hidden state; the state after the last token summarises the message; classify from it. For the forecast, feed weeks one by one and predict the next.
The recurrent state has to carry the decisive sentence from paragraph one through every subsequent token to the end. Each step transforms the state, and the gradient that would teach the cell to preserve that information passes back through every one of those transformations — shrinking or blowing up along the way (Vanishing and Exploding Gradients). Long tickets are misrouted because the model effectively forgot the beginning.
- The recurrent state has to carry the decisive sentence from paragraph one through every subsequent token to the end. Each step transforms the state, and the gradient that would teach the cell to preserve that information passes back through every one of those transformations — shrinking or blowing up along the way (Vanishing and Exploding Gradients). Long tickets are misrouted because the model effectively forgot the beginning.
- Training is sequential by construction: step t needs the state from step t−1. A GPU that could process thousands of positions at once waits on a chain of dependent operations, so training long sequences is slow in wall-clock terms no matter how much hardware is added.
- For the forecast, the failure is different — the recurrent model fits three years of weekly counts about as well as a seasonal baseline and worse than it on the launch spikes, at many times the engineering cost. The offline metric looked "fine" only because nobody ran the baseline.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- For routing: predict the team a ticket belongs to from its text. The label is the team that eventually resolved it — which is a proxy, because tickets get bounced.
- For the forecast: predict next week's ticket volume per team from the weekly history. The label is the count that will be observed; the decision is staffing.
- For routing, one example is a token sequence of a few dozen to a few thousand tokens plus a team label. Length varies by an order of magnitude and the decisive sentence can be anywhere.
- For the forecast, one example is a per-team weekly count series, three years long, with strong weekly and annual seasonality and a handful of product-launch spikes.
- Both are "sequences" and that is where the similarity ends: one is discrete symbols where meaning depends on distant context, the other a short numeric series whose structure is mostly trend and seasonality.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An RNN cell computes
h_t = f(W_h · h_{t−1} + W_x · x_t): the new state is a function of the previous state and the current input, with the same weights at every step. The whole sequence is summarised in a fixed-size vector, and information from early steps survives only if each subsequent update preserves it. - The gradient of a loss at step T with respect to the state at step t is a product of T−t Jacobians of
f. When the relevant singular values are below one the product vanishes; above one it explodes. LSTM and GRU cells add gates — learned sigmoid multipliers that decide what to keep, write and expose — so the cell can carry a value through many steps with the gradient passing almost unchanged along the "keep" path. Gating makes long dependencies learnable; it does not make them cheap, and the sequential dependency remains. - A transformer removes the recurrence. Each position's representation is computed from all positions at once through self-attention (Self-Attention), so the decisive sentence in paragraph one is one attention hop from the final token rather than three thousand state updates away. Every position is computed in parallel, which is why training saturates a GPU; the price is that attention over n positions costs n² and order must be added back explicitly (Positional Information).
A state carried step by step
The recurrent idea is one cell applied repeatedly: read a token, update the state, move on. The state after the last token is the model's summary of the whole sequence. This is compact — one fixed-size vector however long the input — and it is exactly why long dependencies are hard: everything the model will ever know about paragraph one has to survive every update between there and the end.
Backpropagation through those updates multiplies one Jacobian per step. Over a few dozen steps the product is manageable; over a few thousand it is either vanishingly small or enormous, and the parameter update for "remember the request in paragraph one" is either zero or noise. Gating helps by giving the state a path that is nearly an identity — keep what you kept — so a value and its gradient can travel without being transformed every step.
1# h_t = tanh(W_h @ h_prev + W_x @ x_t), same W_h at every step2# dL/dh_t = dL/dh_T * prod_{k=t+1..T} (dh_k / dh_{k-1})3# = dL/dh_T * prod_{k=t+1..T} diag(1 - h_k**2) @ W_h4#5# each factor has norm roughly |W_h| * (1 - h^2) — typically < 16# after (T - t) = 3000 factors: (0.9)**3000 ≈ 1e-137 -> vanished7# (1.1)**3000 ≈ 1e124 -> exploded8# a gate that learns to be ~1 on the "keep" path makes the factor ~1,9# which is the whole trick behind LSTM / GRUThe product form is the point. Gating does not remove the multiplication; it lets the network learn factors close to one along the path it needs. Attention removes the product altogether by making every position one step from every other.
What the transformer changed
Attention replaces "carry the state forward" with "look at everything at once". For the final token to use paragraph one, it computes a relevance score against every position and takes a weighted sum — one layer, one hop, no chain of updates in between. The gradient to paragraph one is a single factor, not a product of thousands.
The second consequence is computational. Nothing in that weighted sum depends on the previous position having been processed, so all positions are computed together as one matrix multiplication — a shape a GPU executes at full occupancy (GPU Fundamentals). That is what made pretraining on web-scale corpora feasible, and pretraining is most of why transformers are what a team can actually download.
Routing accuracy on held-out tickets improved, with almost all of the gain in the longest quartile of tickets.
Bounce rate fell for long tickets and rose for a small group of very long tickets that the model had never seen the length of.
- 1The transformer resolves the long dependency the RNN could not, which is the improvement the offline slice showed.
- 2Tickets longer than the trained context are truncated or run at positions the positional scheme never saw, and the model degrades on them — a failure the offline set, capped at the same length, could not contain.
- 3The resolving-team label is a proxy; some of the "gain" is the model matching the historical bounce pattern rather than the correct team.
Path from token 1 to the output: 3,000 sequential updates. Training step: 3,000 dependent operations, however many GPUs you own. Memory: one fixed-size state.
Path from token 1 to the output: one attention hop per layer. Training step: one batched matrix multiply over all positions. Memory: a 3,000 × 3,000 attention matrix per head per layer.
Constant path length is what makes the long-ticket dependency learnable, and the parallel computation is what makes training at scale affordable. The cost is the n² attention matrix, which is why context length is a budget rather than a setting.
The forecast that did not need any of this
The demand forecast is also a sequence problem, and it is a different one. Three years of weekly counts is 156 numbers. Their structure is a weekly pattern, an annual pattern, a trend and a few spikes. A seasonal-naive model — "next week looks like the same week last year, scaled by the recent trend" — captures most of it in two parameters and is trivially explainable to the staffing manager.
Neural sequence models earn their cost on forecasting when there are thousands of related series, rich covariates, or histories long enough to learn from; on one short series they overfit the spikes and lose to the baseline. "Transformers beat RNNs on text" is true and has nothing to say about this (Forecasting, Forecast Evaluation).
What structure in the sequence decides the output?
when A short numeric series where trend and seasonality explain most of the variance; one or a few series; the consumer needs to understand the forecast.
cost Cannot use rich covariates or learn across many series; a launch spike is an outlier it cannot anticipate.
when Many related series with covariates — promotions, holidays, prices — and tabular-shaped history.
cost Feature engineering for lags and windows must be reproduced at serving time; horizon handling is manual.
when Strict per-step streaming latency, a fixed memory budget, or a small corpus with no pretrained model available.
cost Long dependencies are hard to learn; training is sequential and slow; rarely the best quality today.
when Discrete sequences where distant context decides the output — text, code, long event logs — and a pretrained model exists.
cost Quadratic attention over length; a context-length limit; needs an accelerator; positional scheme may not extrapolate.
How to build it
Most important first.
- For text where meaning depends on distant context, a transformer — usually pretrained — is the default: the path length between any two positions is constant, and the parallel computation is what makes pretraining at scale possible (Transformer Fundamentals).
- For a short numeric series with seasonality, run the seasonal baseline first. Recurrent and attention models earn their cost when there are many related series, rich covariates or long histories; a single three-year weekly series is not that (Forecasting, Baselines Are Mandatory).
- Treat recurrence as a concept to understand, not a first choice to reach for. Where it survives — streaming inference with strict per-step latency, tiny models on constrained devices — it survives because of the fixed-size state, and that property should be the reason you choose it.
- Validate with time respected: whichever sequence model you pick, a random split of tickets by date or of weeks leaks the future (Time-Based Split, Time-Series Validation).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- For routing, accuracy against the first-resolving team, sliced by ticket length. The slice is the number that maps to the decision: the recurrent model's weakness shows only in the long-ticket bucket (Evaluation Slices).
- For the forecast, error against a seasonal-naive baseline on the last held-out year — the ratio, not the absolute number. A model that does not beat the baseline has not earned its cost (Beating the Baseline).
- Do not compare the two models by training loss. The recurrent model's training loss falls fine; its failure is on dependencies its gradient never propagated.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The context that decides the label lies within the sequence length the model was trained on; a longer production input is out of distribution for either architecture, in different ways.
- The per-step latency and the sequence lengths in production match what the architecture was chosen for — an attention model over a sequence ten times longer costs a hundred times more.
- For the forecast, the seasonal structure the baseline captures persists; the learned model's advantage, if any, was measured against that baseline on a held-out period and is re-checked as periods pass.
- Offline: evaluate the routing model on held-out tickets bucketed by length and by where the decisive sentence sits; a model that only works when the request is early is not solving the problem.
- Online: shadow the new router next to the old one for a few weeks and compare the bounce rate — tickets re-routed by humans — which is the outcome the resolving-team label only approximates (Shadow Deployment).
- Over time: keep the seasonal baseline running next to the forecast model permanently and alert when the model stops beating it on the trailing quarter; that is cheaper than any drift detector and answers the question directly.
What can go wrong
- The transformer routes long tickets well and now misroutes tickets that exceed its trained context length, because it was never trained on positions that far in and its positional representation does not extrapolate (Positional Information).
- The team switches the forecast to a transformer too, on the argument that it beat the RNN on text. On a single short series it overfits the launch spikes, and the seasonal baseline nobody ran would have won.
- The routing label — the team that resolved the ticket — changes meaning when teams are reorganised, and both models degrade at once. This is concept drift, not an architecture problem (Concept Drift).
- Attention buys constant path length and parallelism at quadratic cost in sequence length, so the context window is a budget rather than a free parameter (Inference Cost).
- A transformer needs an explicit positional scheme and, in practice, pretraining; a recurrent model has order for free and can be trained from scratch on a small corpus, and that is still sometimes the right trade.
- Simpler forecasting models are cheaper to run, explain and monitor; their cost is that they cannot use rich covariates or learn across many related series, which is precisely where the neural models start to pay.
- "Transformers replaced RNNs, so RNNs were a mistake." Gated recurrence solved the vanishing-gradient problem well enough to power a decade of translation and speech systems. Attention won on parallelism and path length at a scale that made pretraining practical, not because recurrence was wrong.
- "It is a sequence, so it needs a sequence model." A weekly count series is a sequence; so is a sentence. The first has structure a two-parameter seasonal model captures and the second has structure that depends on tokens thousands of positions apart. The word "sequence" is doing no work.
- "The RNN misroutes long tickets, so we need more training data." More data does not lengthen the gradient path the cell can learn across. The fix is architectural — gating, attention or truncation with a smart choice of what to keep — not a bigger corpus.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- SIMPLIFIEDRNN, LSTM and GRU are described at the level of "a state carried step by step" and "gates that decide what to keep"; the actual cell equations, bidirectional variants and attention-augmented RNNs are left out because the gradient-path argument does not depend on them.
- DATA-SPECIFICThe advantage of a transformer is specific to long discrete sequences where distant context decides the output. For short numeric series a seasonal or linear model is usually the right first model, and for many related series with covariates a gradient-boosted model on lag features often beats both neural families.
- CONTESTEDA serious position holds that the transformer's quadratic attention is a temporary state of affairs and that modern recurrent and state-space models, which keep a fixed-size state and linear cost per token, match transformer quality on many tasks while being far cheaper at long context and at inference. That is a real research direction with real results; as of now the pretrained models a team can actually obtain are overwhelmingly transformers, which decides the practical choice.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Signal processing and classical time-series statistics — ARIMA and exponential smoothing are the baselines a neural forecast has to beat, and their assumptions about stationarity are the ones a learned model quietly inherits.