AcceleratorsMODEL-SPECIFICSIMPLIFIEDSCALE-SPECIFIC

Memory Bandwidth & VRAM

What fits on the device is parameters times bytes per parameter, plus activations, plus — for training — optimizer state. What runs fast is bounded by how quickly those bytes can be read, and for large models every token reads all the weights.

Target & data What to measure What must stay true

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

Will this model fit on the device, and once it fits, is inference bounded by arithmetic or by reading the weights?

The problem

A team wants to self-host a seven-billion-parameter language model for an internal assistant. They have a device with twenty-four gigabytes of memory. The engineer says it will fit "because the file is fourteen gigabytes"; the first attempt runs out of memory at the second request, and the second attempt runs but generates text far more slowly than the vendor benchmark.

The obvious approach

The weights file is fourteen gigabytes and the device has twenty-four, so it fits with ten to spare. Load it, serve it; if it is slow, the device must be underpowered — get a faster one.

Why it breaks

The weights are fourteen gigabytes. Activations, the framework's workspace, and the key-value cache for a long context on two concurrent requests take the rest, and the second request fails allocation.

How it breaks — usually after the offline metric looked fine

The weights are fourteen gigabytes. Activations, the framework's workspace, and the key-value cache for a long context on two concurrent requests take the rest, and the second request fails allocation.
Once it fits, each token requires reading every weight once — fourteen gigabytes per token, per sequence, at batch size one. At the device's memory bandwidth that read alone sets a ceiling on tokens per second, and the arithmetic units are nearly idle.
The "faster" device the team wanted has more compute and similar bandwidth; it would barely help. The vendor benchmark was fast because it batched many sequences, so each weight read produced many tokens.
Nothing about this was visible from the file size, and utilisation reported the device as busy — it was busy reading.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

The surrounding model generates a response token by token; the serving system's target is that the model fits in device memory with headroom for the working set, and produces tokens fast enough for an interactive session at the expected concurrency.
The decision is which device, which precision and what batch size — three knobs that all trade against each other through memory.

Data

The model has seven billion parameters stored in sixteen-bit floats: two bytes each, so fourteen gigabytes of weights before anything else is allocated.
Each generated token requires a forward pass whose activations — and the cache of attention keys and values for every previous token — live on the device for the duration of the request, and scale with context length and concurrent requests.
The vendor benchmark was measured at a batch of many concurrent sequences on a device with several times the memory bandwidth.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

Device memory holds four things during inference: the parameters (count × bytes per parameter), the activations of the current forward pass, any per-request cache (for a transformer, the keys and values of every prior token in the context), and framework workspace. Training adds gradients (same size as parameters) and optimizer state — for Adam, two more copies — plus activations for every layer kept for the backward pass.
Memory bandwidth is the rate at which the device can read its own memory. A forward pass reads every parameter at least once. For a large model at small batch, the time to read the weights exceeds the time to multiply them, so the pass is memory-bound: tokens per second is roughly bandwidth divided by weight bytes, and no amount of extra compute helps.
Batching amortises the read. If eight sequences are decoded together, each weight read serves eight tokens, and throughput rises nearly eightfold until the arithmetic units become the bound — the point where the pass becomes compute-bound. Batching is limited by memory too: each sequence's cache must also fit.
Bytes per parameter is the lever that touches both: halving precision halves the weights that must fit and halves the bytes read per token. That is why Quantization is a memory story before it is a compute story.

The fit arithmetic

Whether a model fits is a sum, and every term is knowable before the first allocation. For inference: parameters times bytes per parameter, plus activations for one forward pass, plus the per-request cache times the number of concurrent requests at their maximum context, plus a margin. For training, add gradients and optimizer state, and activations for every layer since the backward pass needs them.

The training column explains why fine-tuning a model that "fits for inference" fails immediately: with Adam the parameter-related memory roughly quadruples before activations, and activations for backpropagation dwarf those for inference.

What has to fit, in bytes

1def inference_bytes(params, bytes_per_param, layers, hidden, ctx, concurrency, kv_bytes=2):
2    weights = params * bytes_per_param
3    # attention cache: keys + values, per layer, per token, hidden width each
4    kv_per_token = 2 * layers * hidden * kv_bytes
5    cache = kv_per_token * ctx * concurrency
6    activations = 0.05 * weights          # illustrative margin for one forward pass
7    return weights + cache + activations
8 
9def training_bytes(params, bytes_per_param, activation_bytes):
10    weights = params * bytes_per_param
11    grads = weights                       # one gradient per parameter
12    adam_state = 2 * weights              # first and second moment per parameter
13    return weights + grads + adam_state + activation_bytes
14 
15# 7e9 params at 2 bytes: 14 GB of weights before a single token is cached
16# 32 layers x 4096 hidden x 2 (K,V) x 2 bytes = 0.5 MB per token of context
17# 4k context x 4 concurrent = 16k tokens x 0.5 MB = 8 GB of cache
18# -> 14 + 8 + margin does not fit a 24 GB device with headroom

The cache term is the one that grows with product decisions — context limit and concurrency — rather than with the model. It is also the one nobody computes from the weights file.

Every token reads all the weights

Once the model fits, its speed at small batch is set by bandwidth. Generating one token means one forward pass, and a forward pass reads every parameter once. At fourteen gigabytes of weights on a device that reads, say, a few hundred gigabytes per second, that read takes tens of milliseconds — and the multiply-adds it feeds take far less. The units wait on memory.

This is the arithmetic-intensity story from GPU Fundamentals in its sharpest form. Batching eight sequences reads the weights once for eight tokens; the read takes the same time and produces eight times the output. Throughput scales with batch until compute becomes the bound or the cache runs out of memory, whichever first.

Batch (concurrent sequences)	Weight bytes read per step	Tokens produced per step	Bound	Memory for cache
1	all weights	1	memory bandwidth	one context
8	all weights	8	memory bandwidth, approaching compute	eight contexts
32	all weights	32	compute, if the cache fits	thirty-two contexts — often it does not

Bytes per parameter is the lever on both

Halving the bytes per parameter halves the weight term in the fit sum and halves the bytes read per token. It is the only knob that improves capacity and speed together, which is why quantization is the first thing anyone does to a large model and why it needs its own quality evaluation (Quantization).

The assumption the whole design rests on is that the working set stays under capacity at production traffic. That is not a property of the model; it is a property of the model plus the context limit plus the concurrency, and the last two are product settings that change without an ML review.

must stay trueThe working set fits at production traffic

Weights plus per-request cache at maximum context and target concurrency plus workspace stays under device memory with the stated margin.

holds when Context limit and concurrency are enforced at the API, the margin was computed for the maximum rather than the median, and precision has not changed since the fit was checked.

breaks when The context limit is raised for a feature; concurrency is increased for throughput; a longer system prompt is added to every request; a framework upgrade changes workspace size.

how you would know Peak memory against capacity on a dashboard; allocation failures; spill-to-host events; achieved batch size falling as the cache grows.

respond Recompute the fit; then reduce context, concurrency or precision, or evict idle caches — and re-run the quality evaluation if precision changed.

How to build it

Most important first.

Do the arithmetic before choosing the device: parameters times bytes, plus the cache for the maximum context at the target concurrency, plus a margin for activations and workspace. If it does not fit with headroom, reduce precision, context, concurrency, or the model.
Identify the bound: at the target batch size, compare bytes read per token against the device bandwidth and arithmetic per token against its compute. Choose hardware by the one that binds.
Batch concurrent sequences to amortise weight reads, within the memory budget the cache allows; the batch size is set by memory, not by preference.
Reduce the per-request cache where the product allows: shorter contexts, eviction of idle sessions, sharing the prefix cache across requests with the same prompt.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

Peak device memory at the target concurrency and context, against capacity, with the margin stated. This is the number that decides whether the model fits; the weights file size is not.
Tokens per second per device against the bandwidth ceiling (bandwidth divided by weight bytes); if you are near the ceiling, more compute buys nothing and more bandwidth or fewer bytes does.
Compute utilisation is the number the dashboard offers; on a memory-bound workload it is low by construction and does not mean the device is wasted.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

The maximum context length times the maximum concurrency, times the per-token cache size, plus weights and workspace, stays below device memory with a stated margin — checked against production traffic, not the benchmark.
The workload remains memory-bound at the production batch size, so the device was chosen for bandwidth; a change in batch size or model can flip the bound and invalidate the hardware choice.
The precision the weights are stored and read in is the precision the quality evaluation was run at.

How to verify — offline, online, and over time

Offline: allocate at maximum context and target concurrency and record peak memory; measure tokens per second at batch sizes one through the memory limit and plot against the bandwidth ceiling.
Online: alert on peak memory approaching capacity and on allocation failures; track achieved batch size and tokens per second per device.
Over time: re-run the fit arithmetic when the model, precision, context limit or concurrency target changes — any one of them can push the working set over.

What can go wrong

Failure modes in production

Memory fits at the benchmarked context and fails in production when a user pastes a long document, because the cache scales with context and the margin was computed for the median.
Batching is increased for throughput and the cache for the extra sequences crowds out workspace; the framework starts spilling to host memory and throughput collapses.
Precision is reduced to make the model fit and nobody re-evaluates quality on rare inputs; the model fits, runs fast, and is wrong on the cases the assistant was bought for.

What the recommended approach costs

Batching for bandwidth amortisation raises per-token latency for each sequence and needs memory for every sequence's cache; the throughput gain is bounded by memory before it is bounded by compute.
Lower precision fits more and reads faster at a quality cost that concentrates on rare inputs and must be measured on slices.
Shorter contexts and cache eviction reduce memory and reduce what the model can attend to; the product decides whether that is acceptable.

Misreads

"The weights file is smaller than the device memory, so it fits." Weights are the floor. Activations, cache and workspace at production concurrency are the rest, and they are what run out.
"Utilisation is high, so the device is the bottleneck — get a faster one." A faster device with the same bandwidth is the same speed on a memory-bound pass. Read the bound before buying.
"Just use a bigger model." A bigger model reads more bytes per token on the same bandwidth, so it is slower in proportion — before any quality gain is measured. The fit arithmetic and the bandwidth ceiling say what "bigger" costs.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

MODEL-SPECIFICThe per-token cache and the "every token reads all the weights" property are specific to autoregressive transformers; a convolutional classifier has no cache and reads its weights once per image, so its bound at small batch is launch overhead rather than bandwidth.
SIMPLIFIEDThe fit arithmetic ignores fragmentation, framework overheads and the exact cache layout, which can add a meaningful margin; the byte counts and any throughput figures here are illustrative of the shape and not measured on any device.
SCALE-SPECIFICFor models of a few hundred megabytes on a modern device none of this binds — everything fits and batching is about launch overhead; the memory and bandwidth bounds become the whole story at billions of parameters.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Computer Architecturememory-hierarchy bandwidth-bound-workloads latency-vs-bandwidth model-memory-and-quantization main-memory

Cloudgpu-infrastructure