Correctnessimplementation

Compilers and Security

The compiler is a trusted component that can delete your security code, exploit your undefined behavior into a vulnerability, or be malicious itself. The canonical case is a `memset` that zeroes a password buffer being removed as a dead store — which is why `explicit_bzero` and `SecureZeroMemory` exist.

The question

In what ways is the compiler part of my threat model rather than part of my toolbox?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute
What the program is here

The build as an attack surface: a source tree, a toolchain binary, a set of dependencies and a set of transformations, producing an artifact whose relationship to the source is exactly as trustworthy as every component in that list. The security question is not what the source says but what the artifact does, and the two are connected only by components you are trusting.

What this phase may assume or do

Every transformation in this lesson is licensed by the same rule as any other: an optimization may remove or reorder work whose effect is not observable under the language’s semantics. The security-relevant consequence is that "an attacker can read this memory later" is not part of that observability model in C or C++, so a store whose value is never read again is dead by the language’s definition even when it is the entire point of the code. Making it survive requires moving it inside the observability model — a volatile access, a memory barrier, or a function the compiler is not permitted to reason about.

Key points

  • Four distinct problems: compiler bugs, legal removal of security code, undefined behavior compiled into a vulnerability, and a hostile toolchain.
  • A secret-scrubbing memset is a dead store under the language’s observability model, and removing it is legal — which is why explicit_bzero and SecureZeroMemory exist.
  • "An attacker reads this memory later" is not observable behavior in C, so the language cannot express the requirement and the library function has to.
  • The volatile-cast idiom works on current compilers and is not guaranteed; a specified function is a specification, an idiom is a bet.
  • Dereference-then-check makes a null check provably dead, which is why kernels build with -fno-delete-null-pointer-checks.
  • Toolchain trust extends past the compiler to the linker, plugins, the build system and every dependency that executes code at build time.

Four ways the compiler is in your threat model

These are genuinely different problems with different mitigations, and conflating them is how discussions of this topic go nowhere. Take them one at a time.

The first is miscompilation as a security bug: a compiler bug produces code that does not implement your logic, and the logic in question happens to be an authorization check. Rare, and covered by [[miscompilation]]. The second is optimization removing security-critical code — legal, common, and the subject of the next section. The third is undefined behavior in your program being compiled into a vulnerability, which is how a great many real CVEs are produced. The fourth is the toolchain itself being hostile or compromised, which is [[trusting-trust]] and the supply-chain problem.

Four problems, four mitigationstypical
ProblemWhat happensMitigation
MiscompilationA compiler bug turns a correct check into incorrect codeSanitizers, differential testing, pinned and validated toolchains
Optimized-away security codeA legal transformation removes a scrub, a check or a timing-equalizing operationMake the effect observable: explicit_bzero, volatile, barriers
UB compiled into a vulnerabilityA null check is removed because the pointer was already dereferenced; a bounds check is removed because overflow is undefinedRemove the UB; -fno-delete-null-pointer-checks, -fwrapv, sanitizers in CI
Hostile toolchainThe compiler, a plugin, a linker or a dependency inserts code you never wroteReproducible builds, diverse double-compiling, pinned and attested artifacts

The dead store that was the whole point

implementationWhich spelling survives is implementation-specific and has changed over time. explicit_bzero is available on the BSDs and glibc; SecureZeroMemory is the Windows equivalent; memset_s is C11 Annex K and optional, so many toolchains simply do not have it. The volatile-cast idiom works on current GCC, Clang and MSVC and is not guaranteed by any standard, which means it is a bet on optimizer behavior rather than a specification — exactly the kind of bet that stops paying after a compiler upgrade.

Here is the canonical case and it is worth knowing exactly. A function copies a password or a key into a local buffer, uses it, and calls memset(buf, 0, sizeof buf) before returning so the secret does not linger in memory that will be reused, swapped, or captured in a core dump. The compiler observes that buf is a local whose address does not escape and which is never read after the memset. A store whose value is never read is a dead store. It removes it. The secret stays in memory, and the security property the programmer wrote code for is silently absent from the binary.

Nothing has gone wrong from the compiler's point of view, and this is the important part. The language's observability model says what a conforming program can detect about its own execution. "An attacker reads this stack memory after the function returns" is not in that model — it is not a behavior of the program at all. So the store is dead by definition, and dead-store elimination is legal. The mismatch is between the language's notion of observable and the security engineer's.

The fix is to move the operation inside the observability model. That is precisely what explicit_bzero (BSD, glibc), SecureZeroMemory (Windows), memset_s (C11 Annex K) and Rust's zeroize crate exist to do — each is defined or implemented such that the compiler may not remove it. The folk remedies are worse than they look: a volatile pointer cast works in practice on mainstream compilers and is not guaranteed by the standard; a memory barrier after the memset works by making the compiler assume the memory is observed; and calling memset through a volatile function pointer works by defeating the analysis rather than by expressing the requirement.

Dead-store elimination on a secret scrub — legal, and a vulnerability
Before
void check(const char *input) {
  char key[32];
  load_key(key);
  use(key, input);
  memset(key, 0, sizeof key);   /* scrub the secret */
}
After
void check(const char *input) {
  char key[32];
  load_key(key);
  use(key, input);
  /* store removed: `key` is a non-escaping local
     that is never read after this point */
}
Legal only when

Legal whenever key is a local whose address does not escape the function, the stored values are never read on any path afterwards, and the object is not volatile. Under those conditions no conforming program can observe the difference, so removing the store preserves observable behavior as the language defines it.

Illegal when

Illegal if the buffer is volatile, if its address escapes to a function the compiler cannot analyse, or if the scrub is performed by a routine the implementation is required not to elide — explicit_bzero and SecureZeroMemory are specified precisely so that this transformation may not be applied. The security requirement is unchanged in every case; what changes is whether the language’s model can see it.

Undefined behavior as a vulnerability factory

The second large category is your own undefined behavior being turned into an exploitable condition by an entirely legal optimization. The mechanism is always the same: undefined behavior licenses the compiler to assume something, and the assumption removes a check you were relying on.

The best-known shape is the null check that follows a dereference. If a pointer has already been dereferenced, then in C it cannot have been null on any defined execution, so a subsequent if (p == NULL) is provably false and the branch is dead. Code that dereferences first and checks second — a very easy mistake in a long function or after an inlining — loses its check entirely. The Linux kernel accumulated real vulnerabilities of exactly this shape, which is why kernels build with -fno-delete-null-pointer-checks.

The same mechanism produces removed overflow checks, removed bounds checks, and array accesses hoisted above the test that guarded them. The mitigation ladder is worth knowing in order: remove the undefined behavior (best); constrain the language with flags such as -fwrapv and -fno-strict-aliasing and -fno-delete-null-pointer-checks (real cost, real benefit); and add runtime checking — -fsanitize=undefined in test builds, -D_FORTIFY_SOURCE=2 and stack protectors in shipped ones.

  • Dereference-then-check: the check is provably dead and disappears. Mitigate with -fno-delete-null-pointer-checks and by checking first.
  • Self-comparing overflow checks: a + b < a on signed types is undefined and gets folded away. Use __builtin_add_overflow or compare against the type limit.
  • Strict aliasing violations: type-punned reads get reordered relative to writes. -fno-strict-aliasing, or memcpy, or a union where the language allows it.
  • Infinite loops without side effects are undefined in C++ and may be removed entirely, which turns a deliberate hang into a fallthrough.
  • The sanitizers are the only tool that reliably tells you which of these you have. Run them in CI, not once.

Trusting the toolchain itself

The last category is the compiler as an adversary rather than as an over-eager optimizer. [[trusting-trust]] is the deep version — a compiler that inserts a backdoor and also inserts the backdoor-insertion into any compiler it compiles, so the malicious behavior survives even though it appears in no source anywhere. The practical version is broader and much more likely: a compromised toolchain download, a malicious compiler plugin, a build-time code generator pulled from a package registry, a linker script, or a dependency whose build script runs arbitrary code on your machine.

The defences are all about breaking the "trust because we always have" cycle. Reproducible builds — [[reproducible-compilation]] — let independent parties compile the same source and compare artifacts bit for bit, so a compromised builder is detectable by anyone who rebuilds. Diverse double-compiling detects the trusting-trust attack specifically by building the compiler with a second, unrelated compiler and comparing. Pinned, checksummed and attested toolchain artifacts, plus hermetic builds that cannot reach the network, reduce the number of places a substitution can happen.

Two things are worth saying plainly. First, this is not paranoia: real supply-chain attacks have shipped through build systems and through dependency registries, and build-time code execution is a standard feature of most package ecosystems. Second, the compiler is only one node — the linker, the assembler, the build system, every plugin, and every dependency that runs code at build time are all in the same trust boundary, and securing only the compiler secures very little.

How it works

The steps, in the order the compiler takes them.

  • The optimizer computes that a stored value is never subsequently read on any path and that the object cannot be observed elsewhere, and removes the store.
  • The optimizer derives a fact from an undefined-behavior rule — this pointer is non-null, this addition does not overflow — and folds any test of that fact to a constant, deleting the guarded branch.
  • Inlining brings a check and a dereference into the same function, at which point an assumption that was invisible across the call boundary becomes available and removes the check.
  • A hostile build component injects code into the artifact, or substitutes a dependency, at a point where no source review would show it.
  • A secure alternative works by placing the operation outside what the optimizer may reason about: a volatile access, a barrier, or a function the implementation is required not to elide.
  • Reproducible builds detect injection by making two independent builds of the same source bit-for-bit comparable.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • A key stays in stack memory after the function that scrubbed it returns, and appears in a core dump, a swap file or a heap-spray read months later.
  • A null check visible in the source is absent from the disassembly, and a null dereference becomes a controllable crash rather than a caught error.
  • A constant-time comparison is compiled into an early-exit loop, and the timing side channel the code was written to avoid is reintroduced by the optimizer.
  • A build upgrade changes optimizer behavior and silently removes a mitigation that a volatile idiom had been relying on, with no diagnostic anywhere.
  • A dependency’s build script exfiltrates credentials from the CI environment, and nothing in the source tree records that it ran.
  • A binary differs from what the source would produce, and nobody can tell because the build is not reproducible and there is nothing to compare against.

When it helps

  • Any code handling secrets in memory: keys, passwords, tokens, decrypted plaintext. The scrub problem is universal and the fix is one function name.
  • Kernel, driver and embedded work, where undefined behavior is common, the consequences are memory-safety failures, and the mitigation flags are standard practice for good reason.
  • Cryptographic implementation, where the optimizer is an active adversary to constant-time properties the language cannot express at all.
  • Any build pipeline that pulls a toolchain or dependencies over a network, which is nearly all of them.

When it hurts

  • As a reason to disable optimization globally. The performance cost is real, the security benefit is small, and it substitutes a blunt instrument for the two or three specific measures that actually apply.
  • When it becomes cargo cult: volatile sprinkled across a codebase, barriers with no stated requirement, a rule against inlining. Each hides a symptom and none states a property.
  • In threat models where the compiler is genuinely not the weakest link, and effort spent on toolchain attestation would have been better spent on the application’s own memory safety.

What it costs

Every one of these is paid by something.

  • Using a scrub function the compiler may not elide buys the security property and costs a real store on a hot path plus, in some implementations, a barrier that constrains surrounding optimization.
  • Constraining the language with -fwrapv, -fno-strict-aliasing and -fno-delete-null-pointer-checks buys predictability and pays in lost optimizations — notably loop analyses that depend on overflow being undefined.
  • Sanitizers in CI buy definitive detection of undefined behavior and cost build time plus a large runtime slowdown, so they cannot be the only configuration you test.
  • Reproducible builds buy independent verifiability of the artifact and cost real engineering: no timestamps, no paths, no environment leakage, deterministic ordering everywhere.
  • Pinning the toolchain buys supply-chain stability and accrues security debt, since the pinned version stops receiving the fixes that were the reason to upgrade.

What else you could do

What a different compiler or language does instead, and when that is better.

  • Use a language and library that make the requirement expressible: Rust’s zeroize crate marks the operation so it cannot be elided, and safe Rust removes the undefined-behavior category almost entirely.
  • Keep secrets out of ordinary memory: hardware security modules, enclaves, or OS key-storage APIs move the problem outside the compiler’s reach entirely, at the cost of an API boundary and often of performance.
  • Verify or validate the compilation rather than trusting it — [[verified-compilers]] and [[translation-validation]] address the miscompilation arm, though neither addresses the dead-store arm, because that transformation is correct.
  • Assembly or intrinsics for the small pieces where the property is not expressible in the source language, which is what serious cryptographic libraries do for constant-time primitives, paying in portability and reviewability.

See it for yourself

The flag, dump or tool that shows you this directly.

  • See the store disappear: compile with gcc -O2 -S or clang -O2 -S and look for the zeroing loop, then repeat with explicit_bzero and observe that it survives. Compiler Explorer makes this a thirty-second experiment.
  • Find the undefined behavior: clang -fsanitize=undefined,address, gcc -fsanitize=undefined, and -Wall -Wextra plus -Wnull-dereference and -Wstrict-aliasing=2.
  • Constrain the language when you must: -fwrapv, -fno-strict-aliasing, -fno-delete-null-pointer-checks. The last is standard in kernel builds and the reason is exactly the dereference-then-check pattern.
  • Harden the artifact: -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fPIE -pie -Wl,-z,relro,-z,now, and check the result with checksec --file=./binary or hardening-check.
  • Toolchain trust: verify checksums and signatures on toolchain downloads, build hermetically with no network access, and compare artifacts against an independent rebuild — the Reproducible Builds project documents the practice and the common sources of divergence.

Plausible wrong readings

Stated the way a confident engineer states them.

  • "The compiler removed my security code, so it is a compiler bug." It is a legal transformation. The language’s observability model does not include an attacker reading memory after the function returns, so the store is dead by definition. Use a function that is specified not to be elided.
  • "volatile makes it secure." volatile prevents elision of the access on mainstream compilers today. It is not a specification, not a synchronization primitive, and not a defence against anything other than the optimizer.
  • "Disabling optimization would fix these problems." It masks some of them at a large performance cost, leaves the undefined behavior in place, and does nothing at all about a hostile toolchain.
  • "We compile from source, so the supply chain is covered." The compiler, the linker, every plugin, the build system and every dependency that runs code at build time are all inside the trust boundary, and none of them is your source.

Misconceptions

The claim, and what is actually true.

Optimizers removing security code is a bug that should be fixed in the compiler.
The transformation is correct under the language’s definition of observable behavior. The gap is between that definition and the security requirement, which is why the fix lives in the library and the specification rather than in the optimizer.
Undefined behavior is a theoretical concern.
It is a routine source of real vulnerabilities: a removed null check turns a defensive test into an exploitable dereference, and the source still contains the check.
A signed toolchain download makes the build trustworthy.
It authenticates one component. Plugins, the linker, the build system and every dependency executing code at build time are inside the same boundary and are usually authenticated far less carefully.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

The compiler can hurt your security in four ways: a bug that miscompiles a check, a legal optimization that deletes security-critical code, an optimization that turns undefined behavior in your program into a vulnerability, and a compromised toolchain that inserts code you never wrote. The famous case is the second: zeroing a password buffer before returning is a store nobody reads, so the compiler removes it. Use explicit_bzero or SecureZeroMemory, which are specified so it cannot.

practical

Concretely: use a non-elidable scrub for every secret in memory; run -fsanitize=undefined,address in CI and fix what it reports; build kernel-like and driver code with -fno-delete-null-pointer-checks and -fwrapv; harden shipped binaries with -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fPIE -pie -Wl,-z,relro,-z,now and verify with checksec; and pin, checksum and hermetically build the toolchain. Then read the disassembly of one security-critical function and confirm the code you wrote is actually in it.

advanced

The unifying observation is that security properties are frequently properties of the *machine state*, and languages define behavior in terms of an abstract machine that deliberately does not model machine state. Secret residue in memory, timing, cache footprint and speculative execution are all outside the model, so a compiler is free to destroy them and does — and no amount of care in the source expresses a requirement the language has no vocabulary for. That is why these properties are bought with library functions with special dispensation, with intrinsics and assembly, or with hardware. The long-term fix is language-level: an effect or attribute system that lets a program state "this store must be performed" or "this code must not branch on this value" inside the semantics rather than around it, which is [[effect-systems]] pointed at a problem the mainstream languages currently cannot express.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

implementationWhich secure-zero facility exists depends on the platform: explicit_bzero on the BSDs and glibc, SecureZeroMemory on Windows, memset_s in C11 Annex K where implemented at all, and zeroize in Rust. Portable code needs a conditional shim, and the volatile-cast fallback in that shim is an idiom that works on current compilers rather than a guarantee.
specThe reason the scrub can be removed is a specification question, not a quality-of-implementation one: the C and C++ abstract machines define observable behavior as volatile accesses, input/output and program termination. Memory contents after a lifetime ends are not observable, so no conforming program can detect the difference. A language that defined secret-scrubbing semantics would not have this problem, and none of the mainstream systems languages does.
typicalThe claim that mainstream optimizers exploit these undefined behaviors describes GCC, Clang and MSVC at -O2 and above, and the exploitation has broadly increased version over version as analyses have improved. Some embedded and safety-oriented compilers deliberately define more than the standard requires; a program that relies on that is correct there and vulnerable when rebuilt with a mainstream toolchain.

If you were asked this in an interview

  • Why does explicit_bzero exist when memset already does the job?
  • A null check in the source is missing from the disassembly. Walk me through how that happened and whether it is a compiler bug.
  • What is inside your build’s trust boundary besides the compiler?

Connections

Securitysupply-chain-securitydependency-securitymalicious-packagescicd-securityartifact-integrity
Domains that do not exist yet
  • DevOps / Production Engineering — Hermetic builds, artifact attestation and pinned toolchains in a CI pipeline
    The operational half of toolchain trust — how the build environment is isolated, how artifacts are signed and where provenance is recorded — is owned there. What this domain contributes is why the compiler is inside the trust boundary at all and what a hostile one can do that source review cannot see.
  • Testing & Reliability Engineering — Sanitizers and hardening checks as standing pipeline stages
    Running undefined-behavior sanitizers continuously rather than once is a testing-practice decision with a cost model attached, and that practice is owned there. Which undefined behaviors turn into vulnerabilities, and by what optimization, is ours.