Debug Infoimplementation

Debug and Release Builds

Two default configurations that bundle several independent decisions together, and the bundling is the problem. Optimization, debug information and assertions are three separate dials, and the right shipped build is very often optimized *with* full debug information, split out of the artifact.

The question

Should I really ship a build with no debug information just because it is the release default?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute
What the program is here

A build configuration: a set of flags that jointly determine the generated code, the metadata emitted beside it, and in some languages the *semantics* of the program. The important observation is that the configuration is a tuple — optimization level, debug level, assertion state, runtime-check state, symbol packaging — and "Debug" and "Release" are merely two named points in that space, chosen as defaults by a tool rather than derived from anything.

What this phase may assume or do

Changing optimization level must not change a well-defined program's observable behavior — that is the [[as-if-rule]], and a program that behaves differently at -O0 and -O2 almost always contains undefined behavior rather than having found a compiler bug. Changing the *debug* level must not change the generated code at all. Changing the assertion or runtime-check state, by contrast, *is* permitted to change behavior, because assert and Rust's overflow checks are defined to be conditional on the build. Those three obligations are different, and conflating them is why "it works in debug" is such a persistent and misleading sentence.

Key points

  • "Debug" and "Release" are named bundles of independent dials: optimization, debug information, assertions, runtime checks and symbol packaging.
  • -g and -O are orthogonal. -g -O2 produces instructions identical to -O2 and adds metadata describing them.
  • The correct shipping configuration is almost always optimized *with* full debug information, split out and archived by build ID.
  • Splitting symbols gives a shipped artifact the same size as a no-debug build while keeping crash reports readable.
  • Assertions and runtime checks are *defined* to differ between profiles; that difference is intended, and an assertion with a side effect is a real bug.
  • Most "works in debug, fails in release" is undefined behavior being exploited by the optimizer, and the first response is a sanitizer run.
  • -Og is the underused middle ground: much faster than -O0, much more debuggable than -O2.
  • -O3 is not automatically better than -O2; larger code can cost more in instruction-cache pressure than the transformations gain.
  • Most projects want three or four configurations named for their purpose, not two inherited from a build tool.

The dials, unbundled

A named build configuration sets several independent things at once. Listing them separately is the whole content of this lesson, because almost every complaint about debug and release builds is really a complaint about a bundle that was never examined.

The dials, and what each one alone actually costs:

  • Optimization level (-O0 … -O3, -Os, -Oz, -Og). Changes the generated code. Costs compile time and source correspondence; buys runtime speed or size. Nothing else on this list is affected by it.
  • Debug information (-g, -g1, -gsplit-dwarf, debug = true). Changes only the metadata. Costs build time, link time and artifact size; buys everything in [[debug-information]]. Does not affect the instructions.
  • Assertions (NDEBUG, debug_assertions, -ea). Changes behavior deliberately — an assertion that fires in one build and not the other is doing its job. This is the dial that makes "works in debug" a real phenomenon rather than a mystery.
  • Runtime checks (Rust's overflow-checks, bounds checking in some languages, _GLIBCXX_ASSERTIONS, -fstack-protector). Also changes behavior, and independently of assertions.
  • Symbol packaging (strip, split, objcopy --only-keep-debug, .dSYM, PDB). Decides where the debug information *lives*, not whether it exists. This is the dial that makes "optimized and debuggable and small" achievable.
  • Link-time optimization and codegen units. Affect optimization scope and therefore both speed and build time — see [[link-time-optimization]].
What each default bundles, and what it coststypical
PropertyDebug defaultRelease defaultOptimized + debug info (recommended for shipping)
OptimizationOff or minimalOnOn
Debug informationFullOften none — this is the mistakeFull, split into a separate file
AssertionsOnOffYour choice; independent of the rest
Build timeFastSlowSlow, plus debug-info generation
Binary sizeLarge — unoptimized code plus full metadataSmallSmall shipped artifact; large archived symbols
Runtime speedSlow, sometimes by an order of magnitudeFastFast — identical instructions to release
Stepping fidelityFaithful: one line at a time, every variable readableNoneDegraded but present — see [[debugging-optimized-code]]
Production crash reportsn/aHex addressesFunction names and line numbers

The thing everyone gets wrong

targetThe objcopy/strip sequence is the ELF and GNU-binutils spelling. macOS produces a .dSYM bundle via dsymutil and strips with strip -S; Windows always emits a separate PDB and the association is a GUID stamped in the executable rather than a debuglink section. The -gsplit-dwarf mechanism is DWARF-specific and has no PDB equivalent because PDBs were already separate. The strategy — build with symbols, ship without, archive by build identifier — transfers everywhere; none of these commands do.

The pattern to break is this: "debug means debuggable, release means fast, so a shipped build has no debug information". Every step of that is a confusion, and the last one is expensive.

-g and -O2 are orthogonal flags and always have been. -g -O2 compiles optimized code and emits the metadata describing it. The instructions are identical to -O2 alone — you can verify that in thirty seconds by diffing the disassembly of two builds. What you get in exchange is a binary whose production crashes symbolicate, whose profiles show line numbers, and whose core dumps are interpretable.

The size objection is real and it is solved by *splitting*, not by omitting. Build with full debug information, then separate the symbols into a companion file and ship the stripped binary. On ELF the sequence is objcopy --only-keep-debug to extract, strip to shrink, and objcopy --add-gnu-debuglink to record the association — or use -gsplit-dwarf so the bulk never enters the link at all. On macOS the .dSYM bundle is already separate and you archive it. On Windows the PDB is always separate and the only question is whether you kept it. In every case the shipped artifact is the same size as a no-debug-info build and you retain the ability to read a crash.

The other half of the fix is archival: symbols are only useful if they can be found later, keyed by build ID, for as long as the binary is deployed. That is the [[symbolication]] pipeline, and skipping it is how a team ends up with a directory of crash reports full of hex.

Optimized, debuggable, and small — the standard sequence
1# Build with both. The instructions are identical to -O2 alone.
2clang -O2 -g -fno-omit-frame-pointer -o app app.c
3 
4# Split the symbols out.
5objcopy --only-keep-debug app app.debug
6strip --strip-debug --strip-unneeded app
7objcopy --add-gnu-debuglink=app.debug app
8 
9# Ship `app`. Archive `app.debug`, keyed by build ID:
10readelf -n app | grep -A1 "Build ID"
11 
12# Or let the compiler keep it out of the link entirely:
13clang -O2 -g -gsplit-dwarf -o app app.c    # debug info lands in .dwo files
14 
15# Rust equivalent, in Cargo.toml:
16#   [profile.release]
17#   debug = true          # emit full debug info in the release profile
18#   split-debuginfo = "packed"

-fno-omit-frame-pointer is worth including deliberately: it costs one register and a small amount of performance, and it makes stack walking work for profilers and crash handlers that cannot or will not parse unwind tables. Several major distributions re-enabled it by default for exactly this reason after a decade of the opposite.

When the two builds genuinely behave differently

It is not an absolute rule that debug and release are behaviourally identical, and knowing the specific exceptions is what turns "works in debug, fails in release" from a mystery into a short checklist.

Assertions are defined to differ. NDEBUG disables assert in C and C++; Rust's debug_assertions gate debug_assert!; Java's -ea enables them. This is intended behaviour. The failure mode is an assertion with a side effect — assert(queue.pop() != null) — which is a real bug the language cannot catch, and which vanishes in release along with the pop.

Some runtime checks are profile-dependent. Rust's integer overflow panics under overflow-checks (on by default in dev, off in release) and wraps otherwise — both are defined behaviour, and they are different. _GLIBCXX_ASSERTIONS adds bounds checking to libstdc++ containers. These are semantics chosen by the build.

Undefined behavior. This is the big one and it is not a difference between the builds; it is a difference in what the optimizer was licensed to assume. Uninitialised memory that happened to be zero at -O0, a use-after-free that happened to land in unreused stack, a signed overflow the optimizer folded away, a strict-aliasing violation, a data race that only manifests once the loads were hoisted — all of these produce a program that "works in debug". The correct response is a sanitizer run, not a bisect of optimization flags. See [[ub-and-optimization]] and [[miscompilation]] for the triage.

Timing and layout. A race that never lost at -O0 loses at -O2 because the timing changed. A stack buffer overrun that overwrote padding now overwrites a live value because the frame layout changed. Neither is caused by optimization; both are revealed by it.

The practical rule that falls out: when a bug appears only in release, run the sanitizers before you touch a flag. And when the difference is genuinely an assertion, that is your program telling you the invariant it checks does not hold — the assertion is the message, not the problem.

The middle grounds worth knowing

Between the two named defaults there are several configurations that are the right answer more often than either extreme.

`-Og` optimizes specifically for a good debugging experience: it applies transformations that do not badly damage source correspondence and skips those that do. It is significantly faster than -O0 and significantly more debuggable than -O2, which makes it the right default for a development build in a project where -O0 is too slow to run the test suite.

`-O2 -g` with split symbols is the shipping configuration argued for above, and in most projects it should simply be what "release" means.

`-O1` or `-Os` are worth measuring rather than assuming. -O3 is not universally better than -O2: it enables more aggressive inlining and unrolling, which increases code size and can increase instruction-cache pressure enough to lose. -Os optimizes for size and sometimes wins on real workloads for the same reason. This is a measurement, not a default — see [[optimization-levels]].

Sanitizer builds are a separate configuration entirely (-O1 -g -fsanitize=address,undefined), too slow to ship and far too valuable to skip, and they belong in CI as their own build rather than as a variant of either default.

The general point: a project usually needs three or four configurations, not two, and naming them for what they are for — dev, test, sanitize, ship — is clearer than inheriting a tool's idea of Debug and Release.

  • dev: -Og -g, assertions on. Fast to build, pleasant to step through, fast enough to run.
  • sanitize: -O1 -g -fsanitize=address,undefined, assertions on. Runs in CI; catches what neither other build can.
  • ship: -O2 -g, symbols split and archived, assertions off but invariant violations logged rather than ignored.
  • Measure -O2 against -O3 and -Os on your real workload rather than assuming an ordering.
  • Keep -fno-omit-frame-pointer in the shipping build unless you have measured that you cannot afford it.
  • Whatever you choose, record the exact flags in the artifact metadata — a crash report without the build configuration is much harder to reason about.

How it works

The steps, in the order the compiler takes them.

  • The optimization level selects a pass pipeline, which determines the generated instructions and how much the source correspondence degrades.
  • The debug level selects how much metadata the backend emits alongside those instructions, without altering them.
  • Preprocessor and profile flags such as NDEBUG and debug_assertions remove or retain assertion and check code before the optimizer ever runs, which is why they change behavior rather than merely performance.
  • The linker gathers debug sections, or on some platforms writes a map that a separate tool follows to build a companion symbol file.
  • A post-link step extracts the debug sections into a separate file, strips them from the binary, and records the association by build ID or debuglink.
  • The stripped binary is deployed; the symbol file is uploaded to a symbol store or artifact repository indexed by that identifier.
  • When a crash occurs, the reporter reads the build ID from the binary or the core dump, fetches the matching symbols, and resolves addresses to functions and lines.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • A production crash report contains only hex addresses, because the release build was compiled without -g and there are no symbols anywhere to match it against.
  • A bug reproduces only in release, a team spends a week bisecting optimization flags, and a five-minute sanitizer run finds a use of uninitialised memory.
  • An assertion with a side effect is compiled out in release, and the operation it was performing silently stops happening.
  • Symbols exist but cannot be matched to the deployed binary, because the shipped build was rebuilt from the same source and the build ID differs.
  • A team disables optimization globally in production to make a crash reproducible, and the service misses its latency budget for a month.
  • A profiler shows a flat unreadable stack in production because the build omitted frame pointers and the profiler cannot parse the unwind tables in that context.
  • The debug build is so slow the test suite cannot run in it, so everyone tests only the release build and nobody notices the assertions have never fired.

When it helps

  • Any software that reaches users, where the difference between a symbolicated crash report and an address list is the difference between fixing a bug and closing it as unreproducible.
  • Performance work, where you need optimized code and line-level attribution simultaneously — which is precisely -O2 -g.
  • Incident response: a core dump from production is only readable with the symbols for the exact deployed build.
  • Any project where developers say "it only happens in production", which is usually a statement about build configuration as much as about environment.

When it hurts

  • Where build time is the binding constraint and full debug information roughly doubles it. -g1 gives line tables only and covers symbolication at a fraction of the cost.
  • In hard size-constrained targets — firmware, a small container image — where even split symbols are a storage and process burden that has to be justified.
  • When the debug build is treated as the reference for behaviour, and code accumulates a dependence on assertions or on unoptimized timing that only fails after release.
  • Where symbols are a disclosure concern and no infrastructure exists to store them privately, in which case building them and losing them is the worst of both.

What it costs

Every one of these is paid by something.

  • Shipping optimized code with debug information buys readable production failures and pays build time, link time, and an archival obligation for every release you deploy.
  • Splitting symbols buys a small shipped artifact and pays the operational requirement to store and index them by build ID for as long as the version is live.
  • Disabling assertions in the shipped build buys performance and pays the loss of the earliest, cheapest signal that an invariant broke — which is why logging the violation instead of asserting is often the better trade.
  • Debug builds buy faithful stepping and pay a runtime cost that is routinely several times slower, sometimes enough that the test suite no longer runs in reasonable time.
  • -Og buys a usable debugging experience at moderate speed and pays a performance level nobody would ship, so it is a third configuration rather than a replacement for either.
  • Higher optimization levels buy transformations and pay code size, compile time and — at -O3 in particular — instruction-cache pressure that can be a net loss on the real workload.

What else you could do

What a different compiler or language does instead, and when that is better.

  • Ship a symbol table only, with no DWARF: function names in stack traces and nothing else, at a small fraction of the size. Often the right point for a size-constrained target.
  • Use -g1 (line tables, no variable information): covers symbolication and profiling, drops the ability to inspect variables, and is dramatically smaller than full -g.
  • Keep a build-ID-indexed symbol server — debuginfod, a Windows symbol server, or an artifact bucket — so nothing at all ships with the binary and everything is fetchable later.
  • In managed runtimes the question largely dissolves: bytecode carries line and local-variable tables, and the JIT registers its frames with the runtime, so stack traces are readable without a build decision — see [[bytecode]].
  • Rely on structured logging and metrics rather than post-mortem debugging for services where you control deployment and can reproduce at will. Legitimate, and it fails exactly when a rare crash matters most.

See it for yourself

The flag, dump or tool that shows you this directly.

  • Verify orthogonality: build twice with -O2 and -O2 -g, then objdump -d --no-show-raw-insn both and diff. Identical instructions is the expected result and the whole argument in one command.
  • See what a binary carries: file ./app reports stripped or not; readelf -S ./app | grep debug lists debug sections; size -A ./app shows what fraction of the file they are; readelf -n ./app prints the build ID.
  • Split and verify: objcopy --only-keep-debug, strip --strip-debug --strip-unneeded, objcopy --add-gnu-debuglink, then confirm gdb ./app still finds the symbols via the debuglink or the build-ID path under /usr/lib/debug/.build-id/.
  • Rust: [profile.release] debug = true plus split-debuginfo = "packed" in Cargo.toml; cargo build --release -v prints the exact rustc invocation so you can see what the profile expanded to.
  • Go: debug information is on by default; go build -ldflags="-s -w" strips it, and go version -m ./app reports the build settings baked into the binary.
  • Find the real difference between two configurations: bloaty --debug-file=app.debug -d compileunits app attributes every byte, and Compiler Explorer with two panes shows the instruction-level difference between flag sets directly.
  • Before blaming optimization for a behaviour difference: clang -O2 -g -fsanitize=address,undefined and run the failing case. It answers the question faster than any flag bisect.

Plausible wrong readings

Stated the way a confident engineer states them.

  • "Release builds cannot have debug information." They can, they should, and the instructions are unchanged. Size is handled by splitting the symbols out, not by omitting them.
  • "It works in debug, so my code is correct and the optimizer is wrong." Far more often the program has undefined behavior that -O0 happened to tolerate. Run the sanitizers before forming the conclusion.
  • "-O3 is the fastest setting." It enables more aggressive inlining and unrolling, which grows code and can lose to -O2 through instruction-cache pressure. Measure on the real workload.
  • "Stripping the binary protects the code." It removes names from the shipped file. The behaviour is unchanged, and you have given up your own ability to read a crash unless you archived the symbols first.
  • "Assertions should stay on in production for safety." Sometimes — but an assert that aborts a service on a recoverable condition is a availability problem. Decide per assertion, and prefer logging an invariant violation to terminating on it.

Misconceptions

The claim, and what is actually true.

Debug info makes the binary slower.
It adds sections that are never loaded during normal execution. It costs file size, build time and link time, and the executed instructions are identical.
There are two build configurations.
There are as many as you define. Most projects benefit from at least a development build, a sanitizer build and a shipping build, and naming them for purpose beats inheriting a tool's defaults.
Turning off optimization is a safe way to work around a bug.
It hides a symptom at a large performance cost and usually leaves undefined behavior in place, which will resurface on the next compiler upgrade.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

Debug and Release are just two preset combinations of flags. Debug turns optimization off, keeps full debug information and enables assertions, so it is slow but easy to step through. Release turns optimization on and — in most tools by default — drops the debug information, so it is fast and completely opaque when it crashes in front of a user. The important thing to know is that those two dials are independent: you can and usually should build optimized code *with* full debug information, then separate the symbols into a companion file so the shipped binary stays small.

practical

Set your shipping build to -O2 -g -fno-omit-frame-pointer, split the symbols with objcopy --only-keep-debug plus strip, and upload the symbol file to wherever your crash reporter or symbol server can find it by build ID. In Rust that is debug = true and split-debuginfo in the release profile. Add a third configuration for sanitizers and run it in CI. And when something fails only in release, run -fsanitize=address,undefined before you touch any optimization flag — that ordering will save you days, repeatedly.

advanced

The deeper reason the bundling persists is that the two defaults encode a false dichotomy between "a build for developers" and "a build for users", and that dichotomy stopped being true once production observability became normal. A shipped binary is now the primary subject of investigation: it is what a continuous profiler samples, what a crash reporter symbolicates, what a core dump comes from, and what an incident is reconstructed against. Under that reality the shipping build needs *more* metadata than the development build, not less, and the only thing it needs to avoid is carrying that metadata in the deployed artifact. The design pattern that resolves it — generate everything, ship the minimum, index the rest by an immutable build identifier — is the same one behind [[symbolication]], behind source map upload, and behind [[reproducible-compilation]], which exists precisely so that the archived symbols can be proved to correspond to the shipped bytes.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

implementationWhat a named profile actually sets is a property of the build tool, not of the compiler. Cargo's release profile enables optimization and disables debug info and debug_assertions by default and all three are independently overridable; CMake's Release, RelWithDebInfo and MinSizeRel set different flag sets again; MSBuild's Release configuration keeps PDB generation on by default, which is why Windows crash dumps are frequently more readable than Linux ones from equivalently-configured projects. Read what your profile expands to rather than assuming.
specThe behavioural differences between profiles are language-defined where they exist. C and C++ define NDEBUG as disabling assert; Rust defines integer overflow as panicking when overflow-checks is on and wrapping as two's complement when it is off, and both are correct behaviour rather than one being a bug. Optimization level, by contrast, may not change a well-defined program's observable behavior at all — so a behavioural difference that is not attributable to one of the defined dials is evidence of undefined behavior in the program.
typicalThe recommendation to ship -O2 -g with split symbols describes current mainstream practice for server and desktop software, and is what major distributions and crash-reporting pipelines assume. It is not universal: deeply size-constrained embedded targets legitimately ship with no debug information at all, and some regulated environments require stripped artifacts. The reasoning transfers even where the conclusion does not — the decision should be about where symbols live, not whether to generate them.

If you were asked this in an interview

  • Should a shipped binary contain debug information? Defend your answer including the size objection.
  • A bug reproduces in release and not in debug. What are the first three things you check, in order?
  • What exactly does -g change about the generated code, and how would you demonstrate it?

Connections

Computer Architectureinstruction-cacheregisters
Performancecpu-profilingincident-debuggingprofiling-basics
Cloudartifact-promotioncontainer-build-pipeline
OS & Networkingdebugging-toolsos-debugging-playbookprocess-memory-layout
Domains that do not exist yet
  • DevOps / Production Engineering — Build profiles as release artifacts: what is deployed, what is archived, and how the two are linked
    The recommendation here — build once with full information, ship the stripped artifact, archive the symbols by build ID — is a release-engineering pattern as much as a compiler one, and it fails operationally far more often than technically. Owning the symbol store and the retention policy belongs there; what the flags actually do belongs here.
  • Testing & Reliability Engineering — Which build configuration the test suite runs against, and what that choice hides
    A suite that runs only in the debug configuration never exercises the code that ships, and one that runs only in release never fires an assertion. Deciding which configurations must be tested, and at what frequency, is a testing-strategy question owned there; why the configurations differ behaviourally is ours.