Speculative Optimization
"This value has been a small integer every time, so compile an integer fast path." The profile is evidence, not proof — which is exactly why the fast path is preceded by a check, and why the whole apparatus of guards and deoptimization exists behind it.
How can a compiler emit code that assumes something it has not proved, and still be correct?
IR in which some operations are *conditionally valid*: a specialized instruction paired with an assumption and an exit. The representation splits what used to be one general operation into a predicate, a fast path valid under it, and an escape — so the program is no longer a single sequence of operations but a sequence of bets with recorded settlement terms. It exists to answer a question the general IR cannot express: what may this code assume, and what has to be true for that assumption to remain sound?
A speculative transformation preserves observable behavior only if three things hold together. The assumption is checked by a guard that *establishes* it — a sufficient condition, not a correlated one — before any observable effect of the fast path. The guard's failure path produces exactly the behavior the general operation would have produced for those inputs, resuming at the correct program point. And any value needed to reconstruct that program point remains recoverable in the optimized frame. Evidence from a profile is never part of this argument: it justifies *choosing* to speculate and contributes nothing to correctness. A speculation whose assumption cannot be cheaply and soundly checked is not a risky optimization, it is an illegal one.
Key points
- Speculation replaces a proof with a measurement plus a check; the measurement chooses the bet and the check makes it sound.
- Correctness is never speculated on. What is being wagered is performance, and the settlement mechanism is correct in both outcomes.
- A fact is speculatable only if it has a cheap, sufficient runtime check — which is why engines speculate on representation and identity and never on semantic properties.
- The guard must be placed before any observable effect of the fast path, or the fallback cannot cleanly redo the general operation.
- Assumptions about global state need a global invalidation mechanism rather than a per-execution guard, because there is nothing local to check.
- The economics are dominated by failure probability, because a failed guard costs a deoptimization rather than a branch.
- Because the rollback path is explicitly constructed, its requirements constrain the optimizer — speculation buys freedom on the fast path and pays for it on the slow one.
Evidence, not proof
A static compiler transforms code when it can prove the transformation preserves behavior. That is the discipline of [[optimization-legality]], and it is why so many desirable transformations are unavailable: proving that a + b is an integer addition requires proving that a and b are always integers, and in a dynamically typed language you generally cannot.
Speculation replaces the proof with a measurement and a check. The compiler observes that this site has added two small integers ten thousand times, emits an integer addition, and puts in front of it a test that both operands are small integers. If they are, the fast path runs and is correct. If they are not, control leaves for a path that handles the general case. At no point is anything assumed *without being checked* — the assumption is checked, cheaply, at run time, once per execution rather than being re-derived per operation.
The word "speculative" is therefore slightly misleading, and it is worth being precise about what is being risked. Correctness is not being risked. What is being risked is *performance*: if the bet is wrong often, the program pays for the check, pays for the exit, and pays for the compile that produced code it cannot use. Speculation is a wager about frequency, settled by a mechanism that is sound either way.
// general: look up "x" on obj, which may involve // walking a prototype chain, running a getter, or a dictionary lookup v = get_property(obj, "x")
guard shape_of(obj) == Shape#42 else deoptimize@offset_9 v = load [obj + 16] // fixed offset for "x" in Shape#42
Only if the shape check is performed before any observable effect, the shape identity soundly implies that "x" is a plain data property at offset 16 for every object of that shape, the runtime invalidates this compiled code if a change could make that implication false — a prototype mutated, a property redefined as an accessor, a shape transitioned — and the interpreter state at bytecode offset 9 is fully reconstructible at the guard. The profile that said the site was monomorphic justifies the choice of Shape#42 and contributes nothing to this argument.
The property could be an accessor whose getter has observable effects, and the guard checks only that the object is "an object with an x" rather than checking the specific shape. Then a load at a fixed offset skips a getter the program was supposed to run, and the difference is not a performance regression but a wrong program. It is equally illegal if the runtime has no mechanism to invalidate this code when someone later redefines "x" on the prototype: an assumption about global state needs a global invalidation hook, not a local guard.
What is worth speculating on
The set of speculatable facts is much smaller than the set of observable ones, and the filter is the one from [[why-runtime-information-helps]]: the fact must have a cheap, sound check. That single requirement explains the entire catalogue of what real engines speculate on, and why it looks so uniform across very different systems.
Representation facts qualify — is this value a small integer, is this object of this shape, is this array's element kind still packed. Each is one comparison. Identity facts qualify — is this call site still reaching this exact function, is this class still the only implementer of this interface. Each is a pointer comparison, or a flag that a global invalidation mechanism maintains.
Semantic facts generally do not. "This list is sorted", "this loop runs fewer than a thousand times", "this string is valid UTF-8" are all things a profile could observe and none of them has a check cheaper than the work being optimized. So they are not speculated on, however consistent the observation. The available checks, and not the available observations, bound what speculation can do.
| Assumption | What it buys | The check | How it fails |
|---|---|---|---|
| Operands are machine integers | Unboxed arithmetic in registers instead of generic dispatch | A tag or range test per operand, hoistable out of loops | A float or a large integer arrives; also overflow of the fast add |
| Object has this shape | A fixed-offset field load instead of a dictionary lookup | One comparison against the shape pointer | A differently shaped object reaches the site |
| Call site reaches this function | Inlining, and everything inlining enables downstream | A comparison of the target, or a class-hierarchy assumption with global invalidation | A second implementation is loaded or a different target arrives |
| This value is never null | A hoisted null check, or none at all | One comparison, once, outside the loop | A null arrives and the guard exits |
| This global is still constantimplementation | Constant folding through a runtime value | No per-execution check at all — a global invalidation record | Someone writes to it, and every dependent compilation is discarded |
| This array stays packed | Bounds-check elimination and direct element access | An element-kind check on entry | A hole is created, or an out-of-range index is written |
The cost structure of a bet
Speculation is worth it when the saving on the common path exceeds the cost of the check, amortized over how often the bet holds. That is a genuine inequality with real terms, and it is why speculation is a compiler decision rather than an always-on transformation.
The check is cheap and not free. [[guards]] covers what it actually costs — the instruction, its effect on scheduling, and the values it forces the compiler to keep alive. The saving is often large: a shape check plus a fixed-offset load in place of a dictionary lookup is perhaps an order of magnitude, and a guarded direct call in place of a virtual dispatch enables inlining and everything inlining unlocks.
The failure cost is where the arithmetic goes wrong. A guard that fails is not merely a branch: it triggers [[deoptimization]], which materializes an interpreter frame, discards the compiled code, updates the profile and drops the activation into a lower tier. That is thousands of times the cost of the guard itself. So the calculation is not "how much does the check cost" but "how much does the check cost, times always, plus how much does failure cost, times rarely" — and the second term dominates as soon as "rarely" stops being rare.
- The saving is per execution of the fast path and is often a large multiple, not a percentage.
- The check is per execution too, but is usually a compare and a branch the predictor learns instantly.
- The failure is thousands of times the check's cost, because it is a deoptimization and not a branch.
- Therefore the decision is dominated by the *probability* of failure, which is exactly what the profile estimates and exactly what can change under you.
- Which is why engines track per-function deoptimization counts and eventually stop speculating in a function that keeps failing — a mechanism that exists because the arithmetic above has no stable answer.
Speculation is not a JIT-only idea
It is worth noticing how much of the stack already works this way, because it makes the pattern feel less exotic and clarifies what is actually novel here. A CPU's own speculative execution runs instructions past an unresolved branch and discards the results if the prediction was wrong: bet, checkpoint, verify, roll back. A database optimizer picks a plan from cardinality estimates and, in some systems, re-plans mid-execution when the estimate proves wrong. An ahead-of-time compiler with [[profile-guided-optimization]] biases layout on measured probabilities.
What distinguishes a JIT's version is that the rollback is *visible in the compiled code as an explicit exit with metadata attached*, and that the thing being rolled back is a whole program state rather than a few in-flight instructions. Hardware speculation rolls back automatically within a fixed window; a JIT must construct the rollback path itself and prove it lands correctly.
That difference has a consequence worth carrying into the next two lessons. Because the rollback is explicit and constructed, its requirements flow backwards into the optimizer. The compiler cannot destroy a value that a rollback path needs, cannot reorder an observable effect past a guard, and cannot place a guard after work the general path would have to redo. Speculation buys freedom on the fast path and spends it on the slow one.
How it works
The steps, in the order the compiler takes them.
- Read the profile at each candidate site and classify the observed facts: operand representations, receiver shapes, resolved call targets, nullness, element kinds.
- For each fact, determine whether a sufficient runtime check exists that is cheap relative to the operation being specialized; discard the ones that do not.
- Emit the specialized operation and precede it with a guard testing exactly that fact, placed before any observable effect of the fast path.
- For facts about global state rather than local values, record a dependency with the runtime instead of a guard, so that a violating change invalidates this compilation.
- Attach to each guard the bytecode offset and state map that a fallback would resume from.
- Optimize the specialized code freely, subject to the constraint that every value a state map names must remain recoverable and no observable effect may move across a guard.
- Hoist guards out of loops where the checked value is loop-invariant, so that the check is paid once per loop rather than once per iteration.
- Record deoptimizations per site and per function, and widen or abandon the speculation when the bet has failed enough times.
How it breaks
What the engineer observes when it goes wrong — not what goes wrong internally.
- A guard checks a proxy for the assumption rather than the assumption itself — "is an object" instead of "is this shape" — and a value that passes takes a fast path that is wrong for it. The program returns a wrong answer with no error and no deoptimization to point at it.
- A guard is placed after an observable effect, so the fallback repeats work already done: a getter runs twice, a counter is incremented twice, or an exception is thrown from a state the interpreter never expected.
- A speculation depends on global state with no invalidation hook, so redefining a function or a prototype property leaves compiled code running with a stale assumption. This is a correctness bug that appears only in programs that mutate at run time.
- The bet is wrong more often than the profile suggested, and the site deoptimizes repeatedly. Throughput collapses to below the unoptimized baseline because every failure costs a frame reconstruction and a discarded compilation.
- An overflow case is forgotten: the guard checks that both operands are machine integers and the fast add wraps where the language specifies promotion. Correct for every input the tests used, wrong at the boundary.
- Guards are hoisted out of a loop for a value that is not actually loop-invariant, so the check passes once and the loop runs on values it was never checked against.
When it helps
- Dynamically typed code, where the general operation is dramatically more expensive than the specialized one and the observed distribution is usually degenerate.
- Virtual and interface dispatch, where a guarded direct call unlocks inlining and everything downstream of it — the single largest structural win available.
- Property and field access on objects with stable shapes, turning a lookup into a load at a fixed offset.
- Hot loops where the guard can be hoisted, so the check is paid once and the loop body runs entirely specialized.
- Any operation whose general form must handle cases that this program never produces — which, empirically, is most operations in most programs.
When it hurts
- Genuinely polymorphic sites, where every bet is a coin flip and the failure cost is thousands of times the check cost.
- Programs that mutate structure at run time — redefining methods, adding properties, loading classes late — because each mutation can invalidate compiled code wholesale.
- Latency-sensitive paths, where a deoptimization landing inside a request is a far worse outcome than uniformly slower code would have been.
- Debugging, because the code being executed does not correspond to the source in any simple way and the correspondence changes when a guard fires.
- Security-sensitive reasoning, where "this has always held" is precisely the assumption an adversary constructs input to violate, and where the fast path and slow path may have observably different timing.
What it costs
Every one of these is paid by something.
- Speculation buys code specialized to the actual workload and pays a guard on every execution plus the memory and optimizer constraints of the state maps that make failure survivable.
- Aggressive speculation buys larger wins on stable code and pays with a steeper cliff: more assumptions means more ways to be wrong and a larger deoptimization when one is.
- Hoisting guards out of loops buys per-iteration speed and pays with a larger deoptimization scope — failing a hoisted guard invalidates the whole loop's worth of assumption, not one iteration.
- Depending on global invariants buys speculation on facts with no cheap local check and pays with a runtime-wide invalidation mechanism and the wholesale discarding of compiled code when an invariant breaks.
- Every speculation buys fast-path freedom and pays in slow-path obligations: values kept alive, effects not reordered, guards placed before rather than after.
What else you could do
What a different compiler or language does instead, and when that is better.
- Prove instead of guess. Whole-program analysis, sealed classes and final methods can sometimes establish the fact outright, and a proven fact needs no guard at all —
[[whole-program-optimization]]. - Remove the uncertainty in the language. Static types and monomorphized generics turn a runtime observation into a compile-time property —
[[monomorphization]], at the cost of code size and compile time. - Optimize the general path instead. A faster generic arithmetic routine or a better property-lookup data structure benefits polymorphic code too, and never deoptimizes.
- Use inline caching without full speculation: cache the resolved target at the site and check it per execution, without recompiling the surrounding code around the assumption —
[[inline-caches]], a smaller bet with a much smaller downside. - Speculate at build time rather than run time, using a profile to bias layout and inlining while emitting code correct without any check —
[[profile-guided-optimization]].
See it for yourself
The flag, dump or tool that shows you this directly.
- V8:
--trace-deoptnames the failed speculation and the bytecode offset; the "deopt reason" strings are effectively a catalogue of what the engine speculates on. - V8:
--trace-icshows shape assumptions forming and being widened, which is the input to the speculation decision rather than its outcome. - HotSpot:
-XX:+PrintCompilationmarks recompilations andmade not entrantevents;-XX:+TraceDeoptimizationon a debug build names the reason and the action. - HotSpot:
-XX:+PrintAssemblywith the disassembler plugin shows the guards as real instructions, which is the most convincing way to see that a speculation is a compare and a branch. - .NET:
DOTNET_TieredPGO=1versus0changes how much dynamic information is available to speculate on, and the generated-code difference is visible in a disassembler. - Our transform explorer at
/compilers/optimizeshows the general legality discipline that speculation inherits — every transformation with its precondition — without simulating a JIT.
Plausible wrong readings
Stated the way a confident engineer states them.
- "The compiler assumes the types and hopes for the best." It checks them. The assumption appears in the generated code as a comparison and a branch, and there is a correct path out the other side.
- "Speculation makes the program probabilistically correct." It makes performance probabilistic. The program's behavior is the same under either outcome of the guard; that is the whole design requirement.
- "If the profile is accurate, no guard is needed." The profile describes what has happened. The guard is what makes what happens next safe, and no amount of past evidence substitutes for it.
- "More speculation is always better because guards are cheap." The guard is cheap; the failure is not. The expected cost is dominated by the failure probability, which is why engines back off from sites that deoptimize repeatedly.
- "Any observed fact can be speculated on." Only facts with a cheap, sufficient check. This rules out most semantic properties, which is why the catalogue of real speculations is short and looks similar across every engine.
Misconceptions
The claim, and what is actually true.
Go deeper
The same idea at increasing depth. Stop wherever it stops being useful.
overview
Instead of proving that two values are integers — which it usually cannot — the compiler notices that they always have been, writes fast integer code, and puts a quick test in front of it. If the test passes, the fast code runs. If it fails, the program goes back to the slow general version. Nothing is ever wrong; the only thing at risk is whether the fast path is used often enough to have been worth writing.
practical
The practical lever is stability. Code where the same types, shapes and call targets recur gets specialized and stays specialized; code that varies pays the check, fails it, deoptimizes and loses the optimization for everyone who shares the site. So: do not add properties to objects after construction, do not pass different types through the same hot function, and be wary of one generic helper that every call site in the system funnels through. When something is unexpectedly slow after a change, --trace-deopt will usually name the exact assumption that stopped holding.
advanced
The deep observation is that speculation converts a legality problem into a placement problem. A static optimizer asks "can I prove this?" and stops when the answer is no. A speculative optimizer asks "where must the check go, and what must remain reconstructible at that point?" — and the answers to those two questions determine what it may do everywhere else. This inverts the usual relationship between the fast and slow paths: the slow path, which almost never executes, dictates the shape of the fast path, which always does. Values must be kept alive because a rollback might name them; effects may not migrate across guards because a rollback must be able to redo the general operation cleanly; guards may not sink below the work they protect. An enormous amount of engineering in production JITs is the effort to make those obligations cheap — compact state maps, guards that can be hoisted, rollback paths compiled out of line — because every byte of obligation is a constraint on the code that actually runs.
How much this depends on
Nothing in this domain is true of every compiler. These say how much.
If you were asked this in an interview
- How can a compiler emit code that assumes something it cannot prove and still be correct?
- What property must a fact have before it can be speculated on? Give me one that qualifies and one that does not.
- Where must a guard be placed relative to the work it protects, and what goes wrong if it is placed later?
- A site deoptimizes on one call in a hundred. Is the speculation worth keeping? What do you need to know to answer?
Connections
- Programming Languages & Runtime Internals — Hidden classes, element kinds and the invalidation machinery that watches for a speculation-breaking mutationHalf of every speculation lives in the runtime: the shapes being compared, the cells whose constness is assumed, and the notification path that discards compiled code when a class is loaded or a prototype is mutated. The compiler emits the guard; the runtime maintains the world the guard is testing.
- Security Engineering — Speculation as an attack surface: adversarial input chosen to force repeated deoptimization, and timing differences between fast and slow pathsA speculative system has an input-dependent performance cliff and two observably different code paths for the same operation. Both are things an attacker can steer — into denial of service by forcing constant recompilation, or into a side channel by timing which path ran. Reasoning about that is a security discipline, not a compiler one.