Whole Programtypical

Feedback-Directed Optimization

PGO and a JIT are the same idea run at different times. Both optimize from measured behavior; the only two things that differ are when the evidence is collected and whether a guard is needed to act on it.

The question

What do PGO and a JIT actually have in common, and what is the real difference between them?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute
What the program is here

A program annotated with observations of its own execution, plus — in the dynamic case only — a record of which observations were relied upon. That second half is the entire distinction. A static compiler consumes the observations and forgets them; a runtime keeps them as assumptions it can check, which is what turns evidence into something it is allowed to act on as though it were proof.

What this phase may assume or do

Evidence about frequency licenses a transformation in exactly two ways, and which one is available is decided by whether the wrong case can be detected and undone. Without a check, a profile may only bias a choice among transformations that are correct on every input — layout, inlining, allocation priority. With a check installed before the transformed code and a path back to a correct general version, a profile may additionally license a transformation that is wrong on inputs the evidence did not cover, because the check catches them before they can be observed. An ahead-of-time compiler has no place to put the undo path, which is not a limitation of its optimizer but of the moment it runs at.

Key points

  • PGO and a JIT are the same idea — optimize from measured behavior — separated by when the evidence is collected and whether a guard is available.
  • A JIT profiles the execution it is optimizing, which is why representativeness is a problem for PGO and not for it.
  • Evidence without a guard may only bias choices among universally legal transformations; evidence with a guard may additionally license transformations that are wrong on unobserved inputs.
  • A static compiler cannot speculate because it has nowhere to put the undo path, not because its analysis is weaker.
  • Deoptimization is what makes the guard meaningful: a state map back from optimized machine state to the abstract state the general version expects.
  • The two compose — a startup profile shortens a JIT's warmup, and an inline cache gives a statically compiled program a guarded call site.
  • The choice is about which resource is scarce: evidence, licence to act on it, or merely good layout.

One idea, two schedules

Set the mechanisms side by side and the family resemblance is complete. Both need to know which code is hot. Both instrument or sample to find out. Both use the answer to decide what to inline, how to lay out code, and where to spend the expensive analyses. Both are useless if the observed behavior is not the behavior that matters. The engineering vocabulary differs — "training run" against "warmup", "profile" against "type feedback" — and underneath, the algorithms answering "is this call site hot" are close relatives.

The differences reduce to two, and everything else follows from them.

When the evidence is collected. PGO collects it before the build, from a different execution than the one that will benefit. A JIT collects it during the execution it is optimizing. That is why PGO needs a *representative* workload and a JIT does not: a JIT's profile is, by construction, the profile of the program that is running.

Whether a guard is available. A JIT compiles into a live process it controls, so it can emit a check — this receiver is still that class, this field still has that shape, no subclass has been loaded — and keep a path back to a general version when the check fails. A static compiler produces a binary that is gone from its hands before the first input arrives. It has nowhere to put the undo path, so it cannot rely on anything that might not hold.

That second difference is what makes them non-interchangeable rather than merely differently scheduled. Everything a static compiler does with a profile, a JIT can also do. The reverse is not true, and the missing capability is not analysis but *retraction*.

The same idea at four points on the timelinetypical
Static, no profilePGOPost-link (BOLT-style)JIT
EvidenceHeuristics onlyA past training runProduction samplesThis execution, continuously
Applies toAll future runsAll future runsAll future runsThe run in progress
Can install a guard?NoNoNoYes
May speculate?NoNoNoYes — because it can deoptimize
Typical uses of the evidence—Inlining, layout, allocation priorityBlock and function layoutAll of those, plus type-specialised code and removed branches
What happens when it is wrongMildly suboptimal codeConfidently wrong layout and inliningConfidently wrong layoutA guard fires and execution falls back
What it paysNothing extraA two-phase build; reproducibilityA post-processing step; symbol toolingWarmup, memory, latency variance

The guard is the whole difference

implementationWhere a given system sits is a design decision, not a language property. HotSpot and V8 keep deoptimization metadata and speculate aggressively; Go compiles ahead of time and consumes a pprof profile without speculating; .NET does both across its AOT and JIT modes; and JavaScriptCore runs four tiers with different speculation budgets. The same source language can be executed by systems in different cells of the table above.

It is worth being concrete, because the abstraction hides how small and how decisive the mechanism is. A JIT observes that a call site has only ever seen one receiver type. It compiles a version specialised to that type — inlined, field offsets fixed, dispatch gone — and emits, immediately before it, a comparison against the expected type. If the comparison fails, control leaves the specialised code and re-enters a general version, reconstructing whatever state the optimized code had rearranged. That reconstruction is [[deoptimization]], and it is only possible because the compiler recorded a map from optimized machine state back to the abstract state the interpreter expects.

Now hold the same observation in a static compiler. It has seen, in a training run, that a call site only ever used one type. It may lay the code out so that type's path is the fall-through, and it may inline the corresponding implementation as an inline cache would. What it may not do is delete the general path, because there is no comparison it can place that has anywhere to go if it fails, and no state map that would let it get back. The evidence is identical; the licence is not.

This also explains why the two techniques compose rather than compete. A runtime with tiered compilation still benefits from a profile gathered before startup — it shortens warmup, which is the JIT's largest weakness — and a statically compiled program still benefits from a JIT-like inline cache at a call site the compiler could not resolve. The evidence and the guard are separate resources, and a system can obtain them from different places.

  • Evidence without a guard — PGO, post-link layout, static branch hints. Biases legal choices. Wrong evidence costs performance, never correctness.
  • Guard without evidence — a defensive type check that always passes, an inline cache that has not warmed. Correct, and buys nothing until observations arrive.
  • Evidence plus guard — speculative inlining, type specialisation, branch elimination, unboxed representations. This is the combination that produces the large wins, and the only one that can be *wrong* at run time rather than merely suboptimal.
  • Neither — the heuristic ahead-of-time compiler. Predictable, reproducible, and leaving real speed on the table for dynamically typed and heavily polymorphic code.

What each one is really buying

Framing them as one family makes the choice between them a matter of which resource is scarce, rather than a matter of taste.

If the problem is that the compiler does not know which code is hot, and a representative workload exists, evidence is the scarce resource and PGO supplies it cheaply. If the problem is that the compiler cannot know the *types* — because the language is dynamic, or the call is behind an interface with many implementations, or the shapes only exist at run time — then no amount of evidence helps without the licence to act on it, and the missing resource is the guard. That is the real reason dynamically typed languages gravitate to JITs and statically typed compiled languages generally do not: it is not that JITs are faster, it is that speculation is the only way to recover information the type system never captured.

And there is a third case that is easy to miss. When the problem is that the hot code is *scattered*, neither evidence nor speculation is the answer — layout is, and a post-link tool applied to an existing binary gets it with the least machinery of all. Recognising which of the three you have is most of the decision.

The same observation, exploited two ways
Before
// observed: this call site has only ever seen Circle
total += shape.Area()          // indirect dispatch through a vtable
After
// STATIC (PGO): bias, no guard — the general path must remain
//   the Circle case is inlined and laid out as the fall-through,
//   and the indirect call is still there for every other type

// DYNAMIC (JIT): guard, then speculate
if (shape.class != Circle) deoptimize()   // <- the licence
total += 3.14159265358979 * shape.r * shape.r
Legal only when

The static form is legal unconditionally: it reorders and duplicates code while leaving the general dispatch reachable, so every receiver type still behaves correctly. The dynamic form is legal only because the guard precedes the specialised code and a correct general version remains reachable through the deoptimization path, with a state map letting the runtime reconstruct the interpreter-level state the optimized frame had rearranged.

Illegal when

The static form drops the general dispatch on the strength of the profile — then a receiver the training run never produced executes the wrong code. Or the dynamic form emits the guard but the runtime cannot reconstruct state at that point, so the fallback resumes with a corrupted frame. Both are miscompilations, and neither is detected by the build.

How it works

The steps, in the order the compiler takes them.

  • Some component observes execution: inserted counters, hardware samples, or a runtime counting call sites and recording receiver types.
  • The observations are aggregated into a hotness or type profile attached to program locations.
  • A decision procedure uses the profile to rank candidates — which call sites to inline, which blocks to place together, which values to specialise.
  • If no check can be installed, only transformations correct for every input are applied, and the profile serves purely as a tie-breaker.
  • If a check can be installed, the compiler emits a guard testing the observed condition and compiles the specialised code behind it.
  • The compiler records, for each guarded point, a map from the optimized frame back to the abstract state a general version would need, so execution can be resumed correctly.
  • When a guard fails, control transfers to the general version through that map, and the runtime usually records the failure so it does not speculate the same way again.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • A team assumes a JIT will recover the performance a stale profile lost, and gets neither: the JIT is warming up on every short-lived process while the static profile misdirects the layout.
  • Speculation succeeds in benchmarks and thrashes in production, because a rare receiver type appears often enough to fire the guard repeatedly and force recompilation.
  • A static build acts on a profile as though it were a proof, deletes a path the training run never took, and produces wrong output on an input nobody tested.
  • A short-lived process pays a JIT's warmup on every invocation and never reaches the tier that would have paid for it, so measured throughput is worse than an unoptimized static build.
  • Deoptimization metadata is incomplete at a guard, and the fallback resumes with a corrupted frame — a bug that appears only under the rare input the guard exists for.
  • Latency percentiles worsen after a JIT tier-up, because compilation happens on a thread competing with the request path, while the mean improves and hides it.

When it helps

  • Choosing between mechanisms: naming which resource is missing — evidence, licence, or layout — settles most of the argument quickly.
  • Reasoning about hybrid systems, where a startup profile, a tiered runtime and post-link layout are all present and it is unclear which one is producing a result.
  • Understanding why dynamically typed languages ended up with JITs and statically compiled ones mostly did not, which is a question about recoverable information rather than about speed.
  • Debugging performance that varies between runs: only the guarded mechanisms can behave differently on identical input, so the variance localises immediately.

When it hurts

  • Treating the framing as a claim that JITs subsume static compilers. Speculation costs warmup, memory and latency predictability, and there are workloads where each of those is disqualifying.
  • Assuming that because both use profiles, a profile from one is usable by the other. The representations, granularity and freshness requirements differ enough that they generally are not.
  • Applying it to code whose time is not in the generated instructions, where all four cells of the table are equally irrelevant.

What it costs

Every one of these is paid by something.

  • Collecting evidence before the build buys a shippable, predictable artifact and pays with the representativeness problem — the evidence describes a different execution than the one that benefits.
  • Collecting it during execution buys perfect representativeness and pays with warmup, memory for the compiler and its metadata, and latency that varies as tiers change underneath a running program.
  • Adding a guard buys the licence to speculate and pays with the check itself on every execution, the deoptimization metadata that must be kept live, and a failure mode where a wrong guess costs a recompilation rather than a few cycles.
  • Refusing to speculate buys reproducible, predictable performance and pays by leaving the largest wins unavailable on polymorphic and dynamically typed code.
  • Combining a static profile with a dynamic runtime buys shorter warmup and pays with two profiling systems to operate and a harder attribution problem when performance changes.

What else you could do

What a different compiler or language does instead, and when that is better.

  • Pure heuristics: no evidence, no guard, fully reproducible. Frequently the right answer, and the baseline every other option must beat — see [[compile-time-vs-runtime]].
  • Post-link layout tools, which take production evidence and apply it to a finished binary. Least machinery, captures the layout portion, cannot speculate.
  • Inline caches inside otherwise statically compiled code, which buy a guarded call site without a whole JIT — see [[inline-caches]].
  • Tiered execution that starts interpreted and promotes on hotness, spending compilation effort only where it is repaid — see [[tiered-compilation]] and [[on-stack-replacement]].
  • Ahead-of-time compilation of a dynamic language with a fallback interpreter for the cases it could not resolve, which trades peak speed for startup and predictability.

See it for yourself

The flag, dump or tool that shows you this directly.

  • JVM: -XX:+PrintCompilation and -XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining show what tiered up and what was inlined; -XX:+TraceDeoptimization shows the guards that failed.
  • V8: --trace-opt, --trace-deopt and --print-opt-code show speculation and its retractions, which is the clearest available demonstration that the guard is real.
  • Clang and GCC PGO: -fprofile-use with the mismatch warnings enabled, to see the static half of the same idea.
  • Compare a static PGO build and a JIT-executed build of comparable programs and watch time-to-peak-throughput — the warmup curve is the visible signature of when the evidence was collected.
  • perf stat across repeated runs of the same input: a guarded system produces run-to-run variance that a static binary does not, which is a diagnostic in itself.

Plausible wrong readings

Stated the way a confident engineer states them.

  • "A JIT is just PGO that happens at run time." It is PGO plus a guard. The guard is what permits transformations PGO can never apply, and it is the entire difference in capability.
  • "A static compiler with a good enough profile could do what a JIT does." It could make the same decisions and not the same commitments. Without a place to put the undo path, speculation is unavailable at any level of analysis.
  • "Feedback-directed means it adapts while running." Only in the dynamic case. PGO is feedback-directed and completely fixed once the binary is built.
  • "Guards are an implementation detail of JITs." They are the licence. Remove the guard and every speculative transformation becomes a miscompilation.

Misconceptions

The claim, and what is actually true.

JIT and PGO are competing techniques.
They supply different halves of the same idea and are routinely combined: a startup profile shortens a JIT's warmup, and a JIT covers the call sites a static profile could not resolve.
The advantage of a JIT is that it can recompile.
The advantage is that it can *retract*. Recompiling is how it acts on that ability; the ability itself comes from the guard and the state map.
PGO is safer because it does not speculate.
It is more predictable. It is not safer in the sense of correctness — both are correct — and a bad profile still produces a genuinely slower binary.
Feedback-directed optimization requires a virtual machine.
Post-link layout tools and instrumented PGO are feedback-directed and operate on ordinary native binaries. The virtual machine is required only for the guarded variety.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

Both PGO and a JIT optimize using measurements of the program actually running. PGO takes the measurement earlier, from a separate run, and bakes the result into the binary. A JIT measures the run it is speeding up, and — crucially — can put a check in front of an assumption and undo it if the check fails. That undo ability is the only thing a JIT can do that PGO cannot.

practical

When someone proposes one of these, ask which resource is actually missing. If the compiler simply does not know what is hot and a realistic workload exists, that is PGO. If the hot code is scattered through a large binary, a post-link layout tool gets most of it for far less machinery. If the compiler cannot know the types at all, no static evidence helps and you need a runtime that can guard and retract. And if the process lives for two hundred milliseconds, discount anything that has to warm up, because it will not.

advanced

The unifying variable is not time but *retractability*, and it is worth carrying beyond compilers. Any system optimizing from observed behavior faces the same fork: commit irreversibly and be limited to changes that are correct under every input, or install a detector and a rollback and become free to act on what is merely likely. Database query plan caching with replan-on-cardinality-miss, a CDN prefetching with a cache-miss path, a scheduler placing work optimistically with migration available — all of them are this. The compiler version is unusually clean because the correctness obligation is stark: get the state map wrong and the fallback resumes with a corrupted frame, which is why deoptimization metadata is among the most heavily tested code in any serious runtime and why the ability to speculate is bought at a much higher engineering price than the speculation itself suggests.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

typicalThe four-cell table describes the mainstream shape and real systems blur it. Some ahead-of-time compilers emit guarded specialisations with a fallback path compiled in, which is speculation without a JIT; some runtimes deoptimize only at a restricted set of points, which limits what they may speculate on. Read the specific system rather than the category.
implementationWhich observations a runtime records, and how aggressively it acts on them, is a per-implementation and per-version decision: HotSpot, V8, JavaScriptCore and .NET differ in tier counts, speculation budgets, deoptimization granularity and how failures are remembered. A behaviour observed in one is not evidence about another, and changes between releases of the same one.
specNothing in this lesson is required by any language specification. A specification constrains observable behavior; whether an implementation reaches that behavior by interpretation, ahead-of-time compilation or guarded speculation is unconstrained, which is exactly why one language can be implemented at several points on this table.

If you were asked this in an interview

  • What can a JIT do with a profile that an ahead-of-time compiler cannot, and why?
  • A profile shows a call site has only ever seen one type. What may each of a static compiler and a JIT do with that, and what does each need in place first?
  • Would you expect a startup profile to help a JIT-executed program, and what exactly would it improve?

Connections

Computer Architecturebranch-predictionspeculative-executioninstruction-cache
Performancejit-and-warmupcontinuous-profilingtail-latencypercentilesmeasure-before-optimizing
Domains that do not exist yet
  • Programming Languages & Runtime Internals — The runtime machinery that makes retraction possible: frame state maps, safepoints, on-stack replacement and code invalidation
    The guard is a compiler-emitted check, but the ability to unwind an optimized frame back into an interpretable one belongs to the runtime. This lesson identifies retraction as the whole distinction; the structures that implement it, and their cost in memory and safepoint frequency, are that domain's subject.