Tiered Compilation
Not one compiler but several, arranged from instant-and-slow to expensive-and-fast, with code promoted upward as it proves hot and demoted back down when a speculation fails. Startup and steady state stop competing for the same knob.
Why do engines run several compilers instead of one good one, and what decides which tier a piece of code is in?
One function exists simultaneously in several forms — bytecode, baseline native code, optimizing native code — plus the profile shared between them and the entry point that decides which form a call reaches. The representation is therefore not a program but a *ladder with a current position*, and it exists to answer a question a single compiler cannot: how do you serve code that must start instantly and also run fast for hours, when those two demands want opposite amounts of compile time?
Every tier must implement the same language semantics, so a program's observable behavior may not depend on which tier is executing it — the tiers are optimizations under the as-if rule of [[as-if-rule]], not variants. Concretely: promotion from tier N to tier N+1 is legal only if the higher tier's code is correct under the guards it inserted; demotion is legal only if the lower tier can be entered at the exact bytecode offset the higher tier exited from, with the abstract machine state of [[vm-state-model]] correctly materialized. A tier that is faster and subtly different is not a tier, it is a bug with a performance win attached.
Key points
- Tiering exists because startup and steady state want opposite compile budgets, and a per-function decision is the only way to serve both.
- A baseline tier is not a weak optimizing compiler; it is a template compiler that removes dispatch and speculates on nothing.
- Lower tiers pay the instrumentation cost that higher tiers consume, so profiling is a tax on the code that has not been optimized yet.
- Only the top tier speculates aggressively, and only the top tier therefore carries deoptimization metadata and its constraints.
- The bottom tier never goes away, because it is both the starting point and the landing site for every failed speculation.
- Promotion is a counter crossing a threshold; demotion is a guard failing, and demotion is what makes the aggressive optimization above it affordable.
- A function that repeatedly promotes and demotes burns compile time and makes no progress, which is a distinct and diagnosable failure.
- Tier names and counts are engine-and-version specific; the ladder structure has been stable far longer than any of the names on it.
One knob cannot serve both ends
Put a single compiler between the program and the machine and you have to choose. Compile cheaply and everything starts fast and runs slowly forever. Compile expensively and everything starts slowly — sometimes unusably so, for a large application — and eventually runs fast. Neither answer is acceptable for a browser, a server or an application platform, because they all need both.
Tiering removes the choice by refusing to make it globally. Code starts in the cheapest tier, and only code that demonstrates it will repay a more expensive compile gets one. The threshold is the only tuning decision, and because it is per function rather than per program, the same system can serve a script that exits in forty milliseconds and a server that runs for a month.
The second reason for multiple tiers is subtler and just as important: a lower tier is where a higher tier goes when it is wrong. Speculation is only affordable because there is somewhere to fall back to, so the interpreter is not merely the starting point — it is the safety net that makes everything above it possible. That is the connection between this lesson and [[deoptimization]], and it is why the bottom tier never goes away.
What each tier is for
The tiers are not "the same compiler with more passes enabled". They have different jobs, and the middle ones exist to fill a specific gap: code that is hot enough to deserve native compilation but not stable or hot enough to justify an optimizing compile that might have to be thrown away.
A baseline tier is a template compiler. It walks the bytecode and emits a fixed instruction sequence per opcode, with no IR, no analysis and no register allocation to speak of. It is typically compiling at a rate close to how fast it can write bytes, and it removes the dispatch loop while keeping every type check — so it buys several times the interpreter's speed for almost no compile time, and speculates on nothing.
An optimizing tier is a real compiler: SSA IR, inlining, escape analysis, [[register-allocation]], and specialization against the profile with guards and state maps. It is expensive, it is the only tier that can deoptimize, and it is applied to a small minority of functions.
| Tier | Compile cost | What it removes | Speculates? | Entered when |
|---|---|---|---|---|
| Interpreter | None — the bytecode already exists | Nothing; it is the baseline of correctness | No | Immediately, and after any deoptimization |
| Baseline / template | Roughly linear in bytecode size, no IR | The dispatch loop and the operand stack traffic | No — every type check remains | After a low invocation count |
| Mid tier | A fast SSA compiler with limited inlining | Redundant checks, some boxing, short call chains | Modestly, with guards | When a function is clearly hot but not yet proven stable |
| Optimizing tier | Expensive: full IR, full pass pipeline | Type tests, calls, allocations, bounds checks | Aggressively, with guards and state maps | After a high count, or from a hot loop via [[on-stack-replacement]] |
| Back down | The cost of a rebuilt frame | Nothing — it restores generality | n/a | On guard failure or invalidation — [[deoptimization]] |
Three real ladders
It is worth seeing the actual names once, with versions attached, because the vocabulary is used loosely and the differences between these systems are real design differences rather than branding.
V8 as of 2024 runs Ignition, a bytecode interpreter; Sparkplug, a non-optimizing baseline compiler that emits code by walking the bytecode with no IR; Maglev, a mid tier with a lightweight SSA IR; and TurboFan, the optimizing tier with full speculation and deoptimization. Sparkplug and Maglev were both added after TurboFan existed, which is the clearest evidence available that the gap between "interpreter" and "optimizing compiler" was too wide to leave open.
HotSpot runs a bytecode interpreter, C1 (the client compiler, fast and lightly optimizing) and C2 (the server compiler, expensive and aggressively speculative), with the interpreter and C1 both collecting profile data for C2. Its levels are numbered rather than named, and code can be compiled by C1 with or without profiling depending on where it is expected to go next. GraalVM can replace C2 entirely, which is a useful reminder that the top tier is a component and not the architecture.
.NET compiles IL with a quick JIT on first call — minimal optimization, fast compile — and recompiles methods that cross a call-count threshold with the full optimizing JIT, using dynamic PGO data collected by instrumented tier-0 code when it is enabled. ReadyToRun images add a fourth possibility: ahead-of-time compiled code that starts fast and can still be replaced at run time by a tier that knows more.
- All three keep the bottom tier permanently. Nothing is ever "fully compiled" in a way that removes the fallback.
- All three collect profile data in a lower tier and consume it in a higher one, which means the lower tier pays the instrumentation cost on behalf of the higher.
- All three added tiers over time rather than removing them, which is the empirical answer to "why not one good compiler".
- Only the top tier speculates aggressively enough to need deoptimization, so only the top tier constrains its own optimizer with state maps.
- The names in this list will be wrong eventually. The structure — cheap tier collecting evidence, expensive tier consuming it, a path back down — has been stable for two decades.
Code moves both ways, and the downward move is the interesting one
Promotion is the obvious direction and the boring one: a counter crosses a threshold, the function is queued, a background thread compiles it, the entry point is patched, and subsequent calls arrive in the new code. A function already executing does not benefit until it returns and is called again — unless the system implements [[on-stack-replacement]], which exists precisely because a long-running loop entered once would otherwise never reach a higher tier.
Demotion is the direction that shapes the architecture. When a guard in optimized code fails, execution must continue in a tier that makes no such assumption, at the exact point the guard was checked, with the abstract machine state materialized. The optimized code is then typically discarded and the site's profile updated, so a recompile does not immediately reinstate the same failed bet.
Between those two lies the pathology worth naming: a function that oscillates. It is promoted, deoptimizes on the second call, is promoted again from a profile that has not learned anything, and deoptimizes again. Engines defend against this with de-optimization counts per function, permanent blacklisting after repeated failure, and profile widening on deopt — and when the defence does not engage, the CPU profile shows a compiler thread pinned at a hundred percent while the application makes no progress.
How it works
The steps, in the order the compiler takes them.
- Execute new code in the cheapest available tier, instrumented with per-function invocation counters and per-loop back-edge counters.
- When a counter crosses the tier-up threshold, enqueue the function for the next tier, usually on a background compiler thread so execution is not paused.
- Compile at the target tier: a template walk of the bytecode for a baseline tier, or IR construction, profile-driven specialization, optimization and register allocation for an optimizing tier.
- Install the result and patch the entry point so subsequent calls reach the new code; existing activations continue in their current tier unless on-stack replacement transfers them.
- Keep the profile alive and shared, so that a function promoted from the baseline arrives at the optimizing tier with feedback already collected.
- On a guard failure in optimized code, materialize the interpreter state from the guard's state map, resume in a lower tier at the recorded bytecode offset, and record the deoptimization against the function.
- Widen or invalidate the feedback that motivated the failed speculation, so the next compile makes a different bet.
- Track repeated deoptimizations per function and stop promoting a function that has failed too often, rather than looping.
How it breaks
What the engineer observes when it goes wrong — not what goes wrong internally.
- A function promotes and deoptimizes in a cycle. The application appears to hang or crawl while a compiler thread runs flat out, and the application's own profile shows nothing unusual because the time is not in application code.
- The tier-up threshold is too high for a workload whose functions are each called a moderate number of times, and the program spends its whole life in the baseline tier while every synthetic benchmark of it looks fine.
- The threshold is too low and a startup path compiles thousands of functions that are executed once, delaying first response by hundreds of milliseconds and inflating memory with code that will never run again.
- A compile queued on a background thread completes after the request that needed it, so the tail of the latency distribution is dominated by requests served in the wrong tier — visible only in the latency percentiles, never in the mean.
- Two tiers disagree about a corner of the semantics — floating-point rounding, an integer edge case, evaluation order — and the program produces different results depending on how many times it has run. This is the failure that makes tier-differential testing mandatory.
- Profiling instrumentation in the lower tier is expensive enough that a program which never tiers up is slower than the same engine with tiering disabled.
When it helps
- Any system that must start fast and also run fast: browsers, application servers, IDEs, anything a person waits for and then keeps using.
- Workloads with a small hot core inside a large codebase, where compiling everything expensively would be almost entirely wasted.
- Deployments where a cold start is on the critical path — serverless functions, autoscaled instances, freshly deployed containers — because the lowest tiers determine what that first second looks like.
- Systems that need aggressive speculation, since a lower tier is a precondition for having somewhere to fall back to.
- Mixed workloads on one runtime, where a per-function threshold gives each function the treatment its own behaviour justifies.
When it hurts
- Latency-critical paths where a tier transition or a deoptimization landing inside a request is worse than uniformly slower code. Some trading and real-time systems deliberately warm every path and then forbid further compilation.
- Memory-constrained deployments, where holding several compiled forms of the same function plus profiles is a real cost.
- Benchmarking, where the number depends on iteration count, warmup and execution order, and comparing two implementations honestly requires controlling all three.
- Debugging and profiling, where a stack sample can land in any of several forms of the same function and symbolication has to know about all of them.
- Short-lived processes, which pay for the machinery, cross no thresholds, and would have been better served by an ahead-of-time compiled binary.
What it costs
Every one of these is paid by something.
- Multiple tiers buy independent control of startup and steady state, and pay in implementation surface: every tier is a code generator that must implement the full language identically, and a semantic difference between two of them is a bug that appears only after N executions.
- Instrumenting the lower tiers buys the profile that makes the top tier worth having, and pays a per-execution cost in exactly the code that has not been optimized yet.
- Background compilation buys uninterrupted execution and pays in CPU contention with the application, plus a queue whose depth becomes a latency variable nobody planned for.
- Keeping several compiled forms of a function buys instant tier transitions and pays resident memory proportional to how much of the program has ever been warm.
- A low tier-up threshold buys faster time-to-fast and pays with compile work on functions that never repay it; a high one buys the opposite. Both directions have a cost, which is why the threshold is tuned per engine and per workload rather than derived.
What else you could do
What a different compiler or language does instead, and when that is better.
- One tier only: interpret and never compile. Small, predictable, portable, and permanently slow on hot code —
[[interpreter-performance]]. - One tier only: compile everything ahead of time. No warmup, no compile surface at run time, and no specialization to this execution —
[[aot-compilation]]. - Ahead-of-time compile plus a JIT that may replace it, which is .NET's ReadyToRun and Android's ahead-of-time DEX compilation with a profile-guided recompile. Startup comes from the static code, steady state from the dynamic one.
- Ahead-of-time compilation from a recorded profile, which recovers a large part of the specialization with none of the runtime machinery —
[[profile-guided-optimization]]. - A tracing JIT, which abandons the method as the unit of compilation entirely and compiles hot loop traces across function boundaries. PyPy and LuaJIT take this route; the tier vocabulary applies loosely and the mechanics differ substantially.
See it for yourself
The flag, dump or tool that shows you this directly.
- HotSpot:
-XX:+PrintCompilationprints a line per compilation including the tier number, andmade not entrantlines mark code being discarded — the demotion events, in order. - HotSpot:
-XX:TieredStopAtLevel=1restricts everything to C1, which is the cleanest experiment for measuring what the optimizing tier is actually contributing to your workload. - V8:
--trace-optand--trace-deopton Node print promotions and demotions with reasons;%GetOptimizationStatusunder--allow-natives-syntaxreports a function's current tier directly. - .NET:
DOTNET_TieredCompilation=0forces the optimizing JIT for everything, which trades startup for steady state and makes the tiering contribution measurable by difference. - Our tier tracker at
/compilers/jitsteps the counters, the threshold crossing and the transition back down on a small program, without claiming any engine's thresholds.
Plausible wrong readings
Stated the way a confident engineer states them.
- "The baseline tier is a worse version of the optimizing tier." It is a different kind of compiler with a different job: emit code as fast as it can be written, remove dispatch, assume nothing. Adding optimization passes to it would make it the wrong tool.
- "Once code reaches the top tier it stays there." Deoptimization moves it back down as a matter of routine, and repeated deoptimization can stop it being promoted again at all.
- "More tiers is always better." Every tier is another full implementation of the language's semantics and another place for the tiers to disagree. Engines add them when there is a measured gap, and the gap has to be large to justify one.
- "Warmup is just the first few calls." Warmup is however long it takes the profile to become representative and the compiles to complete, which for a large server application can be minutes and can be re-triggered by a traffic shift.
- "Tiering is a JIT thing, so ahead-of-time compiled languages do not have it." Anything that can replace code at run time can tier: .NET tiers over ReadyToRun images, and Android tiers over ahead-of-time compiled DEX. The relevant question is whether replacement is possible, not how the first version was produced.
Misconceptions
The claim, and what is actually true.
Go deeper
The same idea at increasing depth. Stop wherever it stops being useful.
overview
Instead of one compiler, an engine keeps several: one that starts instantly and runs slowly, one or two in between, and one that is expensive but produces very good code. Code begins at the bottom and is moved up when it has run enough times to be worth the effort. If the fast code turns out to have assumed something wrong, the code moves back down. Startup speed and long-run speed stop being the same decision.
practical
Two practical consequences. First, measure after warmup and measure warmup separately — a benchmark that runs a function a thousand times is measuring the top tier, and your production code path that runs it forty times per request may never leave the baseline. Second, when performance is inexplicable, ask what tier the code is in before asking anything else: -XX:+PrintCompilation, --trace-opt and --trace-deopt answer that in one line, and a large fraction of "mysterious" slowdowns turn out to be a function that deoptimized once and never came back.
advanced
The architectural insight is that a tier ladder is a control system, not a pipeline. Counters and thresholds are the sensor and the setpoint; promotion and demotion are the actuator; and like any control system it can oscillate. Everything engines do around tiering — deoptimization counts, permanent blacklisting, profile widening on failure, hysteresis in thresholds, background compilation queues with priorities — is damping. The consequence is that tiering behaviour is emergent rather than specified: two workloads with identical instruction counts can land in completely different steady states depending on the order in which their functions became hot. That is why tier-level behaviour must be treated as a measured property of a workload rather than a derived property of the engine, and why performance work on a tiered runtime starts with "what tier is this in and how did it get there" rather than with the generated code.
How much this depends on
Nothing in this domain is true of every compiler. These say how much.
If you were asked this in an interview
- Why would an engine add a baseline compiler when it already has an interpreter and an optimizing compiler?
- What decides when code moves up a tier, and what decides when it moves down?
- A service is fast in steady state and slow for the first minute after every deploy. What is happening, and what are your options?
- What has to be true for two tiers to be interchangeable, and how would you test it?
Connections
- Programming Languages & Runtime Internals — The code cache: where compiled forms live, how entry points are patched, and what happens when it fillsA tier ladder assumes there is somewhere to put several compiled versions of the same function and a cheap way to switch between them. Code cache sizing, eviction and entry-point patching are runtime mechanisms, and when the cache fills the tiering policy silently stops working — a production failure that looks nothing like a compiler problem.
- DevOps / Production Engineering — Warmup as a deployment concern: traffic ramping, pre-warming, and why the first minute after a rollout is a different systemThe lowest tiers decide what a cold instance does under load, so tiering turns an engine internal into a rollout strategy. Load-balancer ramp-up, readiness gating on warmup and capacity planning for cold starts are all consequences of this lesson and are owned there.