What a JIT Costs

The most visible cost is a period during which the program runs slower than it will later, and it is worth separating two things people merge. Startup is the time before the program does anything useful — loading, parsing, initializing. Warmup is the time after that during which the program is doing useful work at less than its steady-state speed, because the profile is not yet representative and the compilations have not yet happened.

Warmup can be long. For a large server application it is routinely tens of seconds and can be minutes, and it is re-entered rather than paid once: a traffic shift makes new code hot, a deployment resets everything, a deoptimization storm can effectively restart it for one function. This is why load balancers ramp traffic to new instances, why readiness checks sometimes gate on synthetic warmup traffic, and why the first requests after a deploy are systematically worse than the rest — an engine internal that has become a deployment concern.

And warmup is a *latency* cost, not a throughput cost, which is why it hides from averages. A service whose mean latency is fine can have a warmup period that dominates its 99th percentile after every rollout. Looking at the latency percentiles rather than the mean is what makes it visible at all.

An ahead-of-time compiler's cost is charged to a build. A JIT's cost is charged to the running process, competing with the application for CPU, and in the worst case competing on the same core as the request that triggered the compilation. Background compilation threads mitigate this and do not remove it: the work is still being done by the machine that is serving traffic.

The queue is the part that surprises people. Crossing a threshold enqueues a compilation; it does not perform one. Under a burst — startup, a deployment, a traffic shift — many functions cross thresholds at once, the queue grows, and functions run in the wrong tier for far longer than the threshold implied. That produces a tail-latency effect with no single slow request to blame, which is exactly the shape of problem that is hardest to diagnose from application-level metrics.

There is also a subtler competition. The compiler thread's working set displaces the application's in the caches, and compiled code being installed invalidates instruction-cache lines. Neither is large, and both are the kind of second-order cost that makes a JIT's benefit on a benchmark larger than its benefit in a busy production process.

A JIT holds more of the program in memory than any other execution strategy, because it holds several forms at once. The bytecode remains, because it is the deoptimization target and the source for future compilations. The compiled code exists, potentially at more than one tier for the same function. The profile data — counters, feedback slots, inline caches — is proportional to the number of sites in the loaded program rather than to the hot subset. And the deoptimization metadata described in [[deoptimization]] is emitted per deoptimization point and can rival the generated code in size.

None of these is reclaimed automatically in a useful way. Code caches are finite and, when they fill, engines stop compiling — silently, in the sense that the application sees a performance change with no error. Profiles for functions that were hot once persist unless something ages them. In a long-running process the memory attributable to the compilation machinery grows with how much of the program has *ever* been hot, which is a different and larger quantity than what is hot now.

This is why a JIT is a poor fit for constrained environments and why the alternatives in those settings are ahead-of-time compilation or plain interpretation. It is also why serverless and container platforms, where memory is directly priced, have driven so much interest in ahead-of-time compilation for languages that historically relied on JITs.

A JIT makes performance measurement genuinely harder, and not by a little. The same code, run twice in the same process, can perform differently because the second run is warm. Two benchmarks in one process interfere: the first pollutes the profile of shared code, and the second is measured on caches specialized for the first. Iteration count changes which tier is measured. Input order changes which shapes a site saw. Even the order of the benchmark methods in the file can matter.

The standard defences all exist because of this: warm up before measuring, run each benchmark in a fresh process, measure many iterations and report a distribution rather than a mean, and use a harness that understands the runtime — JMH for the JVM exists largely to handle exactly these hazards. Benchmarking practice has a long catalogue of the ways this goes wrong; the point here is that a JIT is what makes so many of them possible.

The consequence for engineering practice is a real one. Performance regressions are harder to attribute, A/B comparisons need more samples to reach significance, and a change that looks like a large improvement may be a change in which tier the benchmark reached. On a JIT runtime, "we measured it and it was faster" needs more supporting detail than it does elsewhere.

A JIT writes bytes into memory and then executes them. That is precisely the capability that memory-corruption exploits need, and the defence that has been standard for two decades — W^X, "write xor execute", where no page is simultaneously writable and executable — is a defence a JIT must be granted an exception to.

The exception is not total. Engines narrow it: allocate the region as writable, write the code, flush the instruction cache, then remap the region read-execute before any of it runs, so no page is ever both at once. Some go further with dual mappings — one writable view and one executable view of the same physical pages, with the writable view held only by the compiler thread — and hardware features such as Apple's per-thread MAP_JIT permission switching and ARM's memory-tagging extensions narrow the window further still. The residual risk is real: JIT-region memory is a well-known target, and JIT spraying — arranging for attacker-chosen bytes to appear inside generated code — is a documented technique that engines defend against by randomizing constants and code layout.

The practical consequence is that some platforms simply refuse. iOS forbids JIT compilation for ordinary applications, which is why JavaScript engines there run in interpreter or ahead-of-time modes for embedded content; some game consoles, many embedded targets, and hardened server configurations do the same. In those environments a JIT is not a slow option, it is not an option, and the language implementation must have an ahead-of-time or interpreted path — see [[aot-compilation]] and [[compiler-security]].

Set out plainly, the accounting says a JIT wins when a process lives long enough to amortize compilation, when the hot set is small and type-stable, when memory is not the binding constraint, and when the platform permits it. Reverse any of those and the answer changes.

A command-line tool that runs for eighty milliseconds pays the whole machinery and collects nothing. A serverless function invoked once per cold container is the same case, at scale, which is exactly why ahead-of-time compilation and snapshotting have become mainstream for that deployment shape. A memory-priced container pays for code caches and profiles it would rather spend on the heap. A hard real-time path cannot tolerate a compilation or a deoptimization landing inside a deadline. And a locked-down platform will not run one at all.

The honest summary is that a JIT is an excellent answer for long-running, type-stable, memory-comfortable workloads on permissive platforms, and a poor one outside that envelope. Which is worth stating as directly as the wins, because the technique is frequently presented as strictly better than ahead-of-time compilation and is not — the two make opposite bets about process lifetime, and both bets are sometimes right.