On-Stack Replacement
A function that was entered once and has been looping for a minute cannot benefit from being compiled, because nothing will call it again. On-stack replacement swaps the running activation itself over to optimized code mid-loop, which means translating a live frame from one code version's layout into another's.
A function has been running one loop for thirty seconds. Compiling it will not help, because it will never be called again — so how does it ever get faster?
A live activation exists in two incompatible descriptions at once: the interpreter's abstract state at a bytecode offset — instruction pointer, operand stack, slots, as [[vm-state-model]] defines it — and the optimized code's concrete frame, where those same values live in machine registers, in stack slots at compiler-chosen offsets, or nowhere at all because they were folded away. On-stack replacement is a *translation between those two descriptions* at one designated point, and it exists to answer the question a method-entry transition cannot: how does control get into new code when no call is going to happen?
A transfer at a loop header is legal only if the optimized code entered at that point computes the same subsequent observable behavior as continuing to interpret would have. Concretely: the OSR entry point must correspond to a specific bytecode offset that is a loop header; every value live at that offset must be recoverable from the interpreter frame and installable into the location the compiled code expects; the compiled code must not assume anything about work done before entry that the interpreted prefix did not actually do; and any guard the compiled code relies on must be checked on entry rather than assumed, because the prefix ran without them. If a value the optimized code needs was never materialized in the interpreted frame, the transfer is not merely difficult — it is impossible, and the entry point must be rejected at compile time.
Key points
- A long-running loop entered once cannot be reached by entry-point patching, which is how ordinary tier-up works.
- Loop back-edge counters detect the situation; on-stack replacement is the transfer mechanism that acts on it.
- The difficulty is not the decision but the frame translation: the two code versions place the same values in different locations, and some values exist in only one of them.
- The compiler must emit a map from interpreter state at a bytecode offset to the compiled code's expected entry layout, which is compiler metadata like a deoptimization state map.
- Compiling for OSR entry forbids assuming anything the entry prefix would have established, so the OSR version is usually less optimized than a normal compilation of the same function.
- OSR and deoptimization are the same translation problem in opposite directions, and a runtime that implements one has most of what the other needs.
- The transfer is cheap; the speculative compile before it is not, and a loop that ends soon afterwards wastes all of it.
The activation that cannot be promoted
Ordinary tier-up works by patching an entry point: the next call to this function arrives in the new code. That mechanism has a hole in it exactly the size of the most obviously hot code there is. Consider a program whose main reads a file and loops over ten million records. main is entered once. Its invocation counter reads one, forever. Even if the runtime knew perfectly that this is where all the time goes, patching the entry point achieves nothing, because there is no next call.
This is not an edge case. Batch jobs, numerical kernels, game loops, server accept loops, and every benchmark ever written have this shape: one activation, entered once, running for the whole life of the interesting part of the program. A JIT without on-stack replacement is a JIT that cannot optimize the single most common structure of hot code.
The signal is available — [[profiling-and-hotness]] counts loop back edges precisely so that a long-running loop registers as hot without any invocation ever completing. What is missing is the transfer mechanism. OSR is that mechanism, and the reason it is hard has nothing to do with knowing when to do it.
- entryentryentry
called once, long ago
Entering here again is what ordinary tier-up would do, and nothing is going to. - headerloop header↺ loop header
i < n ?
The OSR entry point. Control reaches here ten million times; a compiled version can be entered here. - bodyloop bodylatch
work(i) i = i + 1
Where the counter that triggered all this is incremented. - exitexit
return total
Reached once, at the end. Too late to matter.
- entry→header
- header→bodytrue
- body→header
- header→exitfalse
Read it asEvery mechanism a JIT has for getting control into new code operates on the entry block, and entry is reached once. The back edge from body to header is reached ten million times. On-stack replacement is the observation that a loop header is also a legitimate place to enter a function — as long as you can construct, at that moment, the state the compiled code expects to find.
Why translating a live frame is the hard part
Deciding to transfer is trivial. Performing the transfer is not, and the reason is that the two code versions disagree about where everything is. In the interpreted frame, local i is in slot 3 and the accumulator is on the operand stack. In the optimized version, i is in a machine register, the accumulator has been kept in another, two other locals were proven dead and do not exist, and a third was replaced by a value recomputed on demand. The frames have different sizes, different layouts and different contents.
So the runtime must build the compiled frame from the interpreted one: read each value the compiled code expects, from wherever the interpreter kept it, and write it wherever the compiler decided it should live. That mapping has to be produced by the compiler, because only the compiler knows the target layout — which makes OSR another piece of compiler-emitted metadata, alongside the state maps of [[deoptimization]]. The two are close relatives: deoptimization translates an optimized frame into an interpreted one, and OSR translates an interpreted frame into an optimized one. Same problem, opposite direction, and both constrain what the optimizer may do.
The constraint is real and specific. Compiling a function for OSR entry at a loop header means the compiler cannot assume anything the entry prefix would have established. Values that an ordinary compile would have proven — a null check hoisted out of the loop, a bounds check eliminated because the index started at zero, a type established at function entry — must instead be re-established at the OSR entry point, because the interpreted prefix made no such promises. This is why engines frequently compile a *separate* OSR version of a function rather than reusing the normal one: the entry conditions differ, so the code differs.
| On-stack replacement | Deoptimization | |
|---|---|---|
| Direction | Interpreted frame → optimized frame | Optimized frame → interpreted frame |
| Triggered by | A back-edge counter crossing a threshold | A guard failing, or a runtime invalidation |
| Where it may happen | Only at designated loop headers the compiler prepared | At any guard, which is many points |
| Metadata needed | A map from bytecode-offset state to the compiled entry layout | A map from the compiled layout back to bytecode-offset state |
| Constrains the optimizer by | Forbidding assumptions the interpreted prefix did not establish | Forbidding the destruction of values a state map still describes |
| Frequency | Rare — once per long-running loop | Rare by design, pathological when frequent |
What it costs, and the version that is not worth it
The transfer itself is cheap and happens once. The costs are elsewhere. Compiling an OSR-specific version of a method is compile work that produces code used by exactly one activation and then, typically, discarded — a poor return by the usual accounting, justified only because that one activation is where all the time is.
The code is also usually worse than the ordinary compilation of the same function. Entering mid-loop means the compiler starts with fewer established facts, so it hoists less, eliminates fewer checks and has a less useful view of the values flowing in. Some engines mitigate this by having the OSR version run only until the loop exits and letting a normal compilation take over on the next call, which is a reasonable division: the OSR version rescues this activation, and the ordinary version serves the future.
And there is a real failure case worth stating plainly: OSR into a loop that is about to finish. The counter crossed its threshold on iteration nine hundred thousand of a million, the compile completed at iteration nine hundred and fifty thousand, and the transfer happens with fifty thousand iterations left. The work was done, it was correct, and it did not pay for itself. Nothing in the mechanism can know this in advance, which is why OSR thresholds tend to be higher than method thresholds — the bet is larger, so the evidence required is larger too.
- The transfer is one-time and cheap; the compile that precedes it is neither.
- OSR-compiled code is typically less optimized than the same function compiled for normal entry, because fewer facts are established at a loop header than at a function entry.
- Engines commonly compile a separate OSR version per entry offset rather than making one compilation serve both purposes.
- A loop that exits shortly after the transfer wastes the entire compile, and there is no way to know in advance.
- Without OSR, a runtime either cannot optimize single-activation loops at all, or must exclude looping methods from its lowest tier — which is what .NET did before .NET 7.
- The same metadata discipline that makes deoptimization possible is what makes OSR possible; a runtime that has one is most of the way to the other.
How it works
The steps, in the order the compiler takes them.
- Count loop back edges per function, so that iteration counts inside a single activation register as hotness — see
[[profiling-and-hotness]]. - When a back-edge counter crosses the OSR threshold, request a compilation of that function specialized for entry at the current bytecode offset.
- Compile with the loop header as an entry block, treating every value live at that offset as an incoming parameter and establishing on entry any condition the code will rely on.
- Emit, alongside the code, a map describing for each live value where the interpreter keeps it and where the compiled entry expects it.
- Install the code and mark the running activation so that the next arrival at the loop header takes the transfer.
- At the transfer: read each live value out of the interpreted frame, build the compiled frame with those values in their expected registers and stack slots, and re-check any guard the compiled code depends on.
- Replace the activation record on the stack with the new frame and jump to the OSR entry point; the interpreted frame is discarded, not returned to.
- On loop exit, return through the compiled code's normal return path, and let ordinary tier-up serve any future calls with a normally compiled version.
How it breaks
What the engineer observes when it goes wrong — not what goes wrong internally.
- A runtime without OSR runs a single-activation hot loop interpreted for its entire duration, and a profile shows all the time in one function that the engine never compiled — with the invocation count sitting at one, explaining why.
- The frame translation misplaces a value — right slot, wrong register, or a value the compiler expected to be already unboxed — and the loop continues with corrupted state, producing wrong results from the iteration after the transfer with no error at any point.
- The OSR compilation assumes a fact the interpreted prefix never established, such as an index being in range because a normal entry would have started it at zero. The first post-transfer iteration reads out of bounds.
- The compile completes just before the loop exits, so the work is entirely wasted, repeatedly, in a workload made of many medium-length loops.
- A stack trace taken during the transfer window shows an inconsistent stack — a frame that is neither the interpreted one nor fully the compiled one — which is why the transfer must be atomic with respect to anything that can walk the stack.
- An OSR version is compiled per entry offset and a function with several hot loops accumulates several compilations of itself, each used once, inflating the code cache.
When it helps
- Batch and numerical workloads: one activation, one enormous loop, all of the runtime inside it.
- Benchmarks and warmup harnesses, which almost universally have this shape — and which is why a runtime without OSR looks catastrophically bad on them.
- Server accept loops, game loops and event loops, where the outermost activation is entered once at startup and never returns.
- Any tiering system that wants to put looping methods in a cheap tier at all. Without OSR the alternative is to exclude them, which is what .NET did until .NET 7.
- Startup paths that contain a long initialization loop, where the loop is hot but the enclosing function will be called exactly once in the process's life.
When it hurts
- Workloads made of many medium-length loops, where compiles complete just as loops end and the return on each is negative.
- Memory-constrained runtimes, where per-entry-offset OSR versions of methods accumulate in the code cache alongside the ordinary compilations.
- Situations where the OSR code's weaker optimization matters: a loop that would have been vectorized or fully unrolled given function-entry facts may not be when entered mid-flight.
- Debugging and profiling across the transfer, where a single activation changes representation underneath any tool that was watching it.
- Real-time paths, where the transfer plus the preceding compile is a latency event inside an activation that was already running.
What it costs
Every one of these is paid by something.
- OSR buys the ability to optimize single-activation loops at all, and pays with a separate compilation whose code is typically used by one activation and then discarded.
- Compiling for mid-loop entry buys reachability and pays in code quality: fewer established facts at the entry point means fewer hoisted checks and weaker loop optimization than the same function compiled for normal entry.
- Emitting an entry-state map buys a correct transfer and pays by constraining the optimizer to keep every value the map names materializable at the entry point.
- A higher OSR threshold buys fewer wasted compiles on loops that end soon and pays with a longer stretch of slow iterations before the rescue arrives.
- Supporting several OSR entry points per function buys coverage of functions with multiple hot loops and pays with a compilation and a code-cache entry per point.
What else you could do
What a different compiler or language does instead, and when that is better.
- Do not implement it, and exclude looping methods from cheap tiers so that they are always compiled properly on first call. This is what .NET did before .NET 7: correct, and it gives up tiering exactly where tiering would have helped startup most.
- Do not implement it, and accept that single-activation loops run interpreted. Simple, and it fails on the most common shape of hot code there is.
- Restructure at the language level so the loop body is a separate function, which is then called many times and tiers up normally. This works and it is an unreasonable thing to ask of a programmer, though it is occasionally a real workaround.
- Use a tracing JIT, where the unit of compilation is a hot loop trace rather than a method, so the OSR problem is dissolved rather than solved — entering compiled code at a loop header is the *only* thing a tracing JIT does. PyPy and LuaJIT take this route.
- Compile ahead of time, where the question never arises because there is no lower tier to escape from —
[[aot-compilation]].
See it for yourself
The flag, dump or tool that shows you this directly.
- HotSpot:
-XX:+PrintCompilationmarks OSR compilations with an%and shows the entry bytecode index, so you can see exactly which loop was rescued and when. - HotSpot: run a single-method benchmark with a long loop under
-XX:-UseOnStackReplacementand compare — the difference is the entire content of this lesson, measured. - V8:
--trace-optreports OSR compilations distinctly from ordinary ones; running a script with one long top-level loop and watching the trace shows the transfer in isolation. - .NET:
DOTNET_TC_OnStackReplacement=0disables it, and comparing startup and throughput on a loop-heavy program shows what it is contributing. - JITWatch renders HotSpot OSR compilations alongside normal ones with their entry offsets, which makes the "separate compilation per entry point" design visible.
Plausible wrong readings
Stated the way a confident engineer states them.
- "On-stack replacement means replacing the code on the stack." It means replacing the *activation*: building a new frame in the compiled code's layout from the values in the old one and continuing there. The code was never on the stack.
- "It is just tier-up for loops." The decision is the same; the mechanism is completely different. Ordinary tier-up patches an entry point and waits for a call. OSR translates a live frame between two incompatible layouts.
- "The compiled code is the same, it is just entered elsewhere." Entering at a loop header establishes far less than entering at a function entry, so engines commonly compile a separate, less optimized version specifically for OSR.
- "If a runtime does deoptimization it gets OSR for free." It gets the hard part — the discipline of maintaining a mapping between abstract and concrete frame states — but the direction, the entry-point selection and the compile strategy are all separate work.
- "OSR is a benchmark artifact; real code calls functions." Server loops, event loops, game loops and batch jobs are all single long-lived activations. Benchmarks exposed the problem loudly; they did not invent it.
Misconceptions
The claim, and what is actually true.
Go deeper
The same idea at increasing depth. Stop wherever it stops being useful.
overview
Normally a runtime speeds up a function by making the *next call* go to the compiled version. That does nothing for a function that was called once and has been spinning in a loop ever since — there is no next call. On-stack replacement takes the running invocation and moves it into the compiled version mid-flight, at the top of the loop, by rebuilding its variables in the places the compiled code expects them.
practical
What this means in practice is that a benchmark of one long loop is measuring OSR, and a benchmark of a function called a million times is measuring ordinary tier-up. They are different code paths in the engine and can perform differently, so a microbenchmark shaped as one long loop is not a good predictor for a service shaped as many short calls. If a loop-heavy program is unexpectedly slow, check whether the runtime has OSR at all and whether it fired: -XX:+PrintCompilation marks OSR compilations with %, and the absence of one for your hot loop is the answer.
advanced
The unifying view is that OSR and deoptimization are one capability seen from two sides: the ability to convert between an abstract machine state at a bytecode offset and a concrete machine state in compiled code, in either direction, at designated points. Once a compiler can do that, several other things become possible that look unrelated — debugger-driven "change this variable and continue", hot code replacement of a running method, and moving an activation between machines. Each is the same translation with a different destination. The reason this capability is expensive is that it makes the optimizer answerable to a second observer: not just the program's defined observable behavior, but a metadata description that must remain satisfiable at every designated point. An optimizer that could see only the first is strictly freer, which is precisely why an ahead-of-time compiler can perform transformations a JIT must decline.
How much this depends on
Nothing in this domain is true of every compiler. These say how much.
% in -XX:+PrintCompilation; V8 has moved OSR support between tiers as Sparkplug and Maglev were introduced; .NET shipped OSR in .NET 7 and only then enabled tier-0 compilation for methods with loops. Whether a given runtime has OSR at all, and at which tiers, is a version-specific fact.while loop compiles to a back edge that executes many times inside one activation, visible in the bytecode at /compilers/vm.If you were asked this in an interview
- A function is called once and loops for a minute. How does it ever get compiled, and what makes that hard?
- What metadata does the compiler have to emit for on-stack replacement, and how does it relate to deoptimization metadata?
- Why is OSR-compiled code often worse than the same function compiled normally?
- What would you expect to happen to a loop-heavy benchmark on a runtime with OSR disabled, and why?
Connections
- Programming Languages & Runtime Internals — Stack rewriting: replacing a live activation record atomically with respect to stack walkers, collectors and other threadsThe compiler supplies the map from one frame layout to another; actually performing the swap on a live stack — while a garbage collector might scan it and another thread might request a stack trace — is runtime machinery, and the safety argument for it is entirely about runtime invariants rather than compiler ones.