JITimplementation

Profiling and Hotness

Deciding what to compile is a measurement problem with a cost on both sides: compile too eagerly and you spend time on code that never repays it, compile too late and the program runs slowly through the window where it mattered most.

The question

How does a runtime decide that a piece of code is worth compiling, and what does getting the threshold wrong actually cost?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute
What the program is here

The program is bytecode carrying a set of live counters: one per function for invocations, one per loop back edge for iterations, and a feedback slot per specializable site. Those counters are the representation this lesson works in — a running estimate of *where execution has been spending itself*, maintained cheaply enough to be affordable in the tier that is not yet optimized. They exist to answer a question no static property of the code can: not which function looks expensive, but which one has actually executed.

What this phase may assume or do

Profiling data may drive scheduling decisions freely — anything may be compiled, at any tier, at any moment — because which tier runs a function is not observable behavior. It may drive *semantic* specialization only through the guard-and-fallback discipline of [[speculative-optimization]]. The instrumentation itself carries a precondition of its own: counter updates must not change program semantics, so they may not be observable by the program, must be tolerant of concurrent update from multiple threads, and must not be able to trap. A counter that overflows into undefined behavior or a shared counter with a data race is not a measurement, it is a bug in every program the runtime executes.

Key points

  • Invocation counters and loop back-edge counters answer different questions, and a function called once with a huge loop is invisible to the first.
  • There is no derivable definition of hot: the threshold is a bet that past frequency predicts future frequency.
  • The threshold has a cost in both directions — wasted compilation if too low, a slow window through the period that mattered if too high.
  • Instrumentation is paid in the tier that is already slow, which is the design pressure behind stopping instrumentation once code is optimized.
  • Counters are deliberately non-atomic in most engines: a lost update delays a compile, and an atomic on every call would cost more than the imprecision.
  • Counters must age or decay in a long-running process, or the system eventually promotes code that was hot only during initialization.
  • Sampling costs per unit of time rather than per unit of work and attributes across the stack, which makes it the right tool for humans and the wrong one for tier-up decisions.
  • Deoptimization has to reset or widen the feedback that motivated the failed bet, or the same compile is made again immediately.

Two counters and a sampler

Three mechanisms cover almost all of it, and each answers a different question. Invocation counters are incremented on function entry and answer "is this function called often?". Loop back-edge counters are incremented every time a loop jumps backwards and answer "is this function spending a long time inside itself?" — a question the invocation counter cannot reach, because a function called once with a ten-million-iteration loop has an invocation count of one. Sampling interrupts periodically and records where execution is, which answers both approximately with a cost that does not scale with the number of instrumented sites.

The first two are exact and cost something on every execution; the third is approximate and costs almost nothing per event. Engines use both: counters in the tiers that are already paying an execution overhead, and sampling where the instrumentation would be the dominant cost.

A common composite is to add the two counters. HotSpot's tier-up decision has historically combined invocation and back-edge counts against a threshold rather than treating them separately, which captures "this function is where the program is" regardless of whether that is because of many calls or one long loop.

Three ways to find out where the time is goingtypical
MechanismAnswersCostMisses
Invocation counterIs this function entered often?An increment and a compare per callA single call that runs for a minute
Back-edge counterDoes this function loop a lot?An increment and a compare per iterationWork spread across many short calls
SamplingWhere is execution, statistically?One interrupt per interval, independent of site countAnything rarer than the sample interval; short-lived hot spots
Type feedback slotWhat has flowed through this site?A store and a compare per execution of the siteNothing about hotness — it is a different question
Edge counters in a PGO buildWhich paths did the training run take?Paid entirely at build timeAnything the training workload did not do — see [[pgo-tradeoffs]]

What "hot" means, and why it cannot be derived

implementationThresholds are engine constants, are usually scaled by method size and by which tier is being targeted, and are adjustable by flag. HotSpot exposes -XX:CompileThreshold and the tiered variants -XX:Tier3InvocationThreshold and -XX:Tier4InvocationThreshold; V8 and .NET do not expose comparable stable knobs. The numbers differ between engines, between versions and between 32- and 64-bit builds, so any specific value is a fact about one build rather than about JITs.

There is no principled definition of hot. The honest formulation is economic: a function is hot when the expected saving from compiling it exceeds the cost of compiling it, and every term in that comparison is unknown at the moment the decision is made. The saving depends on how many more times the function will execute, which is a prediction about the future. The cost depends on the function's size and the tier, which is at least knowable. So the threshold is a bet that past frequency predicts future frequency.

That bet is usually good, because programs have loops and loops repeat. It fails in exactly the situations you would expect: at phase boundaries, where a function that dominated startup will never run again; on the very first requests, where nothing has a history; and in workloads whose hot set changes with input, where the history describes the previous input.

Because the threshold is a bet and not a derivation, it is a *tuning decision with a cost in both directions*, and that is the part worth internalizing. Lowering it means compiling functions that execute a few more times and never repay the compile. Raising it means the program runs in a slow tier through a window that may be the only window that matters — the first seconds after a deploy, the first requests of a session. Neither direction is conservative. There is no safe setting, only a choice about which failure you prefer.

  • Too low: compile time spent on functions that run twice; memory filled with code that never runs again; startup delayed by compiling the startup path itself.
  • Too high: the hot path runs interpreted or baseline-compiled through the period that matters; benchmarks that iterate enough will never show it, and production will.
  • Uniform across functions: a large function and a tiny one cross the same threshold, though the compile costs differ by orders of magnitude — which is why engines scale thresholds by size and by tier.
  • Reset on deoptimization: necessary to avoid immediately reinstating a failed speculation, and a mechanism by which a function can be perpetually re-heated and re-compiled.
  • Not decayed: counters that only ever increase eventually promote everything, including code that was hot once during initialization. Ageing or decaying counters is how a long-running process avoids compiling its own history.

The counter is not free, and where it is paid

The instrumentation is a tax on the tier that has not been optimized yet — which is the tier where a tax hurts most, because that code is already the slow code. An increment, a compare and a rarely-taken branch on every function entry and every loop iteration is small in isolation and is multiplied by the same enormous instruction count that makes [[interpreter-performance]] a subject at all.

This produces the design pressure you see in real engines: instrument in the interpreter and the baseline tier, and stop instrumenting once the code is optimized, because the optimized tier is not going to be promoted anywhere. It also explains the sampling alternative — a sampler's cost is per unit of *time*, not per unit of work, so it does not scale with how much the program does.

The subtler cost is what the counters do to the code around them. A counter is memory that must be written on every call, which is a store that cannot be optimized away and, in a multi-threaded runtime, a location several threads write to. Engines generally accept lost updates rather than pay for atomicity, because an approximate counter is entirely adequate for a threshold decision and a contended atomic on every function entry is not.

The shape of an instrumented function entry and loop back edge
1/* on function entry */
2if (++fn->invocations + fn->backEdges > fn->tierUpThreshold)
3  request_compile(fn);           /* queued, not performed here */
4 
5/* on every loop back edge */
6if (++fn->backEdges > fn->osrThreshold && !fn->osrPending)
7  request_osr(fn, bytecodeOffset); /* see on-stack-replacement */

Both counters are plain non-atomic increments, so concurrent calls lose updates. That is deliberate: the value feeds a threshold comparison, an under-count delays a compile slightly, and the alternative is an atomic read-modify-write on every function entry in the program. Note also that neither branch performs the compilation — it enqueues it, so the calling thread is not stopped for a compiler.

Sampling, and what it is good for

typicalMost method-based JITs — HotSpot, V8, .NET — drive tier-up from counters rather than from sampling, and use sampling for other purposes such as HotSpot's stack-walking in the compiler broker. Some systems invert this: Android's ART has used sampling-based profiles written to disk to drive ahead-of-time recompilation between runs, which makes the profile persist across process boundaries. Whether counters or samples drive the decision is an engine design choice, not a property of JITs.

A sampling profiler takes a periodic interrupt and records the current stack. Its cost is fixed per interval rather than per event, its accuracy improves with run length, and it sees the whole stack rather than one function — so it attributes time to callers, which counters cannot do at all.

The tradeoff is bias and blindness. A sampler cannot see anything that happens less often than its interval, systematically misses very short-lived hot spots, and can be skewed by correlation between the sampling clock and the program's own periodicity. It is also, in a JIT, sampling code that may be in any tier and may be inlined into something else, so attributing a sample to a source function requires the same reconstruction metadata that [[symbolication]] and [[debugging-optimized-code]] depend on.

The division of labour that falls out is clean: counters for *decisions the runtime makes about itself*, because they must be exact enough to trigger reliably and cheap enough to run always; sampling for *decisions humans make*, because a human wants attribution across the stack and can tolerate approximation. The two are frequently confused because both are called profiling, and mixing them up produces the belief that an engine's tier-up decision can be read off a CPU profile — a CPU profile is measuring something else entirely.

How it works

The steps, in the order the compiler takes them.

  • Allocate a counter per function and per loop back edge, stored with the function's metadata rather than in the code, so the code can be shared and the counters cannot.
  • Increment the invocation counter on function entry and the back-edge counter on each backward jump, both non-atomically, and compare against the current threshold.
  • When a threshold is crossed, enqueue a compilation request rather than compiling inline, so the executing thread continues.
  • Scale the threshold by the target tier and by the size of the method, so a large function has to prove itself more thoroughly than a small one.
  • Collect type feedback in the same instrumented execution, so that the function arrives at the compiler with both hotness and specialization data.
  • Age or decay counters periodically, so that a long-running process reflects recent behaviour rather than cumulative history.
  • On deoptimization, reset or widen the relevant feedback and record a deoptimization count against the function, so repeated failure eventually stops promotion.
  • Stop incrementing once the function is at the top tier, since there is nothing further to promote it to and the increment is pure overhead.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • A function called once with a long-running loop never crosses the invocation threshold, and the program runs the interpreter for minutes on code that would have compiled in milliseconds — the exact case [[on-stack-replacement]] exists to handle.
  • Counters never decay in a long-lived process, so functions that were hot during initialization are promoted hours later and displace code that is actually hot now.
  • The threshold is tuned against a benchmark harness that iterates ten thousand times, and the production workload calls each function forty times per request, so nothing ever tiers up and the benchmark predicted the wrong system.
  • Counter updates race between threads and consistently lose updates on the hottest, most contended functions — so the hottest code is the last to be promoted, exactly inverting the intent.
  • The compile queue is unbounded and a burst of tier-up requests during startup starves the application of CPU, turning a latency problem into an outage.
  • A function deoptimizes, its counters reset, it re-heats and recompiles with the same profile, and the cycle repeats. CPU is consumed entirely by the compiler and no application profile explains it.
  • A sampling profiler is used to explain a tiering decision, and the attribution is wrong because the samples landed in inlined code that the profiler credited to the wrong function.

When it helps

  • Any system that must choose where to spend a limited compile budget, which is every JIT and also every incremental or cached build system.
  • Long-running services, where counters accumulate enough evidence to be genuinely informative and decay keeps them current.
  • Deciding *not* to compile. A large function with a low count is the clearest possible signal to leave it alone.
  • Diagnosing performance mysteries: the tier a function is in, and the count that got it there, explains a large share of unexplained slowdowns.
  • Feeding an ahead-of-time recompile between runs, as Android does, where the profile is written to disk and used to compile before the next launch.

When it hurts

  • Short processes, where no counter reaches any threshold and the entire instrumentation cost is paid for a decision that is never made.
  • Workloads with sharp phase changes, where the accumulated counts describe a phase that has ended and actively mislead the compiler.
  • Latency-critical first requests, where by definition nothing has a history and the system's worst behaviour coincides with its most visible moment.
  • Highly parallel workloads where counter contention is itself a scalability problem, which is why the increments are unsynchronized and imprecise.
  • Reasoning about performance from a CPU profile alone, where samples are attributed to inlined and tier-varying code and the numbers are honest about time and dishonest about where it belongs.

What it costs

Every one of these is paid by something.

  • Exact counters buy a reliable, deterministic tier-up trigger and pay an increment and a compare on every function entry and loop iteration, in the tier that is already the slow one.
  • Non-atomic counters buy freedom from contention on every call and pay in accuracy exactly where accuracy would matter most — the hottest, most concurrently executed functions.
  • Sampling buys attribution across the stack at a cost independent of workload size, and pays with blindness to anything shorter than the interval and with attribution that inlining makes ambiguous.
  • A low threshold buys a short warmup and pays compile time and memory on functions that never repay it; a high threshold buys the opposite and pays with a slow window that may be the only window the user sees.
  • Decaying counters buy relevance in a long-running process and pay with a risk of demoting something that is genuinely hot but bursty, plus the cost of the decay pass itself.
  • Persisting profiles across runs buys a warm start on the next launch and pays with staleness, storage, and a profile that may describe a different version of the program.

What else you could do

What a different compiler or language does instead, and when that is better.

  • Compile everything on first call. No counters, no thresholds, no instrumentation — and compile time spent on every function the program touches once, which is why systems that do this use a very cheap compiler.
  • Compile nothing and interpret always, which makes the whole question moot at the cost of the whole speedup — [[interpreter-performance]].
  • Take the profile in a separate training run and compile ahead of time from it, moving the entire measurement out of production — [[profile-guided-optimization]].
  • Let the developer annotate. Explicit hints — "compile this eagerly", "never compile this" — replace the measurement with a declaration, which is precise where the developer is right and misleading where they are not.
  • Persist profiles between executions so the next run starts warm. Android does this with ART's profile-guided ahead-of-time compilation between launches, at the cost of storage and of a profile that may not match the current build.
  • Sample instead of count everywhere, accepting approximation in exchange for cost that does not scale with instruction count — the choice a system makes when the instrumented tier cannot afford per-event work.

See it for yourself

The flag, dump or tool that shows you this directly.

  • HotSpot: -XX:+PrintCompilation shows what crossed a threshold and when; -XX:CompileThreshold=N and -XX:Tier4InvocationThreshold=N let you move the threshold and watch the behaviour change.
  • HotSpot: -XX:-TieredCompilation -XX:CompileThreshold=1 compiles nearly everything immediately, which is a fast way to separate "slow because interpreted" from "slow anyway".
  • V8: --trace-opt reports the promotion and, in recent versions, the reason; --interrupt-budget adjusts how quickly functions become eligible, which is the closest thing to a threshold knob.
  • .NET: DOTNET_TC_CallCountThreshold changes the number of calls before a method is queued for the optimizing tier.
  • For human-facing profiling rather than engine decisions, a sampling profiler with a flame graph is the right tool, and it answers a different question from the counters.
  • Our tier tracker at /compilers/jit shows the counters incrementing and the threshold crossing on a small program, with no engine's constants claimed.

Plausible wrong readings

Stated the way a confident engineer states them.

  • "Hot means the function takes a long time." It means the function has executed a lot. A single call that takes a second has an invocation count of one, which is exactly why back-edge counters exist.
  • "The right threshold can be computed." It is a bet on future execution frequency. It is tuned empirically, differs per engine and per workload, and has a cost in both directions.
  • "Profiling in production is expensive." The counters are already running in every JIT you use. A sampling profiler adds cost proportional to the sampling rate, not to the workload, and at typical rates it is a fraction of a percent.
  • "A CPU profile tells me what the JIT decided." It tells you where time was spent, attributed through inlining and tiering in ways that require reconstruction metadata to be right. The engine's own trace flags tell you what it decided.
  • "Counters must be accurate." They must be accurate enough to cross a threshold in roughly the right order. Engines deliberately trade exactness for the absence of synchronization on every call.

Misconceptions

The claim, and what is actually true.

The JIT compiles a function when it becomes slow.
Nothing measures slowness. It compiles a function when a counter crosses a threshold, which is a proxy for frequency, and frequency is a proxy for future benefit. Two functions with identical counts get identical treatment regardless of what they cost per call.
Instrumentation only runs in debug or profiling builds.
The counters that drive tier-up run in every production execution of every mainstream JIT. They are part of how the system decides what to compile, not an optional diagnostic.
A higher threshold is the safe, conservative setting.
It is a different risk, not less risk. It trades wasted compilation for a longer window during which the hot path runs slowly, and for many workloads that window is precisely the part users experience.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

The runtime keeps a tally: how many times each function has been called, and how many times each loop has gone round. When a tally passes a set number, that function is handed to the compiler. There is no clever measurement of how expensive the code is — just counting, on the assumption that what has run a lot will run a lot more. The number that triggers it is chosen by experiment, and both a too-small and a too-large number cost you something real.

practical

Two things to take away. First, if your production code path calls each function far fewer times than your benchmark does, your benchmark is measuring a tier your users never reach — measure with realistic call counts, or check the tier directly with the engine's trace flags. Second, when performance changes for no apparent reason, look at compilation events before looking at your own code: a deoptimization that never recovered, or a function that stopped crossing its threshold after a refactor split it in two, explains a surprising fraction of these.

advanced

The deep point is that hotness detection is an online decision problem with an asymmetric and unknown payoff, and every engine is running a heuristic policy for it without ever writing the objective down. Counters plus a fixed threshold is the simplest policy that works; scaling by method size approximates the cost term; decaying counters approximates non-stationarity; deoptimization counts and blacklisting are the damping that keeps the policy from oscillating. What is striking is how little of this is principled and how well it works anyway — which is itself informative, because it says the underlying distribution is enormously skewed. A tiny fraction of code accounts for nearly all execution in nearly every real program, and when the signal is that strong almost any reasonable detector finds it. The engineering effort accordingly goes not into detecting hotness better but into making the *cost of being wrong* smaller: cheaper tiers to be wrong in, faster compiles to waste, and a fallback path that makes a wrong bet survivable.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

implementationThreshold values, how invocation and back-edge counts are combined, whether counters decay, and how much of this is exposed as a flag are all engine internals and differ between HotSpot, V8, JavaScriptCore and .NET, and between versions of each. HotSpot exposes -XX:CompileThreshold and per-tier thresholds; V8 exposes an interrupt budget with different semantics; .NET exposes a call-count threshold. None of the numbers transfer.
typicalCounter-driven tier-up is what mainstream method JITs do. Tracing JITs count loop headers rather than function entries because their unit of compilation is a trace, and Android's ART has driven ahead-of-time recompilation from sampled profiles persisted between runs. The categories here transfer; which mechanism drives the decision does not.
simplifiedOur VM counts executed instructions for a step budget and does not profile at all — there is no tier to promote to. The counter shapes shown in this lesson are the standard structure rather than output from our simulator, and the tier tracker at /compilers/jit models the decision without generating any code.

If you were asked this in an interview

  • A function is called once and loops ten million times. What does an invocation counter say about it, and what does the system need instead?
  • Why are tier-up counters usually not atomic, and what does that cost?
  • You raise the compile threshold. Name a workload that gets faster and one that gets slower.
  • What is the difference between the profiling a JIT does for itself and the profiling you do with a sampling profiler?

Connections

Computer Architecturebranch-predictioninstruction-cache
Concurrencydata-races
Performancecpu-profilingcontinuous-profilingflame-graphsjit-and-warmuptrace-samplingbenchmark-fallacies
Domains that do not exist yet
  • Programming Languages & Runtime Internals — The compile broker: queues, priorities, background threads, and what happens when compilation cannot keep up with promotion requests
    Crossing a threshold only enqueues work. Whether that work runs, on which thread, at what priority and how it behaves under a burst is the runtime's scheduling problem, and it decides whether a tier-up policy that is correct on paper produces a latency spike in production.
  • Observability & Performance Engineering — Continuous profiling of production workloads, and reading a flame graph across a tiered runtime
    The counters here decide what the engine compiles; a sampling profiler decides what an engineer investigates. They measure different things and are both called profiling, and confusing them leads people to explain engine behaviour with the wrong instrument.