Infratypical

What a Toolchain Actually Contains

Compiler, assembler, linker, loader, debugger and build system are six related but distinct programs with different inputs, different outputs and different failure messages. Knowing which one spoke is most of diagnosing a build.

The question

When my build breaks, which program actually produced this error?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute
What the program is here

A chain of artifacts, each a real file with a documented format, and a different program on each edge. Source text becomes assembly text becomes an object file becomes an executable becomes a process image. The debugger reads a side channel — debug information embedded in the object and the executable — that maps the last of those back to the first, and the build system sits above all of it deciding which edges need traversing at all. Every stage exists as an inspectable artifact on disk, which is the property that makes a build diagnosable rather than mysterious.

What this phase may assume or do

Each program may assume its input is a well-formed instance of the format its predecessor produces, and nothing more. The assembler may assume the assembly is syntactically valid for the target and may not assume anything about the program's meaning. The linker may assume every object file has a valid symbol table and relocation list, and it may assume the One Definition Rule holds without being able to verify it — which is why the mistakes it cannot catch are the dangerous ones. The loader may assume the executable's program headers describe a mappable image. The build system may assume that if its recorded inputs are unchanged, the recorded output is still valid — an assumption only as good as the dependency information it was given.

Key points

  • Preprocessor, compiler, assembler, linker, loader, debugger and build system are distinct programs with distinct inputs, outputs and error vocabularies.
  • Identifying which one produced an error implies the class of fix, and getting it wrong wastes the most time on exactly the confusing cases.
  • Every stage but loading produces a file you can inspect, and stopping the chain early is the most effective diagnostic technique available.
  • Documented intermediate formats are what allow individual stages to be replaced — new linkers, new assemblers, new compilers into the same build.
  • The linker knows mangled symbol names and almost nothing else, so it catches missing and duplicate definitions but not mismatched ones.
  • The build system is not part of the pipeline: it decides which invocations happen, and its failures are about dependency-graph completeness.

Six programs, six vocabularies

The word "compiler" is routinely used for the whole chain, which is convenient right up until something fails. Then the distinction becomes the most useful thing you know, because each program has a characteristic error vocabulary and each vocabulary implies a completely different class of fix.

"No such file or directory" for a header is the preprocessor, and the fix is an include path. "Undeclared identifier" or a type error is the compiler frontend, and the fix is in the source. "Undefined reference to foo" is the linker: the compiler was told foo exists and the linker could not find a definition, which is a build-configuration problem, not a source problem. "Error while loading shared libraries" is the loader, after a successful build, and no change to any source file will affect it. And "nothing to be done for target" is the build system, which has decided the work was unnecessary.

Learning the mapping is worth more than it looks, because the wrong diagnosis wastes the most time in exactly the confusing cases: people edit source for linker errors and rebuild for loader errors, and neither can possibly help.

What each program takes in, what it produces, and how it failstypical
ProgramTakes inProducesCharacteristic failure
PreprocessorSource plus headers and macro definitionsOne self-contained translation unitfatal error: x.h: No such file or directory
CompilerA translation unitAssembly text, or object code directlyType errors, undeclared names, template diagnostics
AssemblerAssembly text for one targetAn object file: machine code, symbols, relocationsunknown instruction or invalid operand for instruction
LinkerObject files and libraries, in command-line orderAn executable or a shared libraryundefined reference, multiple definition, cannot find -lfoo
LoaderAn executable and its dynamic dependenciesA process image with symbols bounderror while loading shared libraries; version mismatch at run time
DebuggerA running process or a core file, plus debug informationA mapping from machine state back to sourceno debugging symbols found; a variable reported as optimized out
Build systemA dependency graph and a set of rulesThe decision of what to rebuild, and the invocationsA stale artifact; or a rebuild of everything for a one-line change

The artifacts are files, and that is the whole diagnostic technique

Every edge in the chain produces something you can look at, and the single most effective debugging technique in this area is to stop the chain early and inspect. -E gives the preprocessed translation unit. -S gives the assembly. -c gives the object file. Each of those is a file with tools that read it: nm and objdump and readelf for objects and executables, ldd for dynamic dependencies, addr2line for turning an address back into a source location.

This is what separates a diagnosable build from a mysterious one, and it is also the reason the classic Unix model has survived several attempts to integrate everything into one program. Separate programs with documented intermediate formats mean any stage can be inspected, replaced or reimplemented independently — which is how lld and mold could appear as drop-in linkers, and how a different assembler or a different compiler can be substituted into the same build.

The one stage with no artifact is the loader, which is why loader problems feel different. There is no file to inspect after the fact; you inspect the executable's recorded dependencies before running it, or you trace the process as it starts. ldd, otool -L, LD_DEBUG=libs and their equivalents exist for exactly that gap — see [[the-loader]].

Stopping the chain at each stage, and looking
1$ gcc -E  main.c -o main.i     # preprocessor output: one translation unit
2$ gcc -S  main.c -o main.s     # compiler output: assembly text
3$ gcc -c  main.c -o main.o     # assembler output: an object file
4$ gcc     main.o -o main       # linker output: an executable
5 
6$ nm -C main.o                 # what this object defines (T) and needs (U)
7$ objdump -d main.o            # disassemble the machine code
8$ readelf -d main              # the executable's dynamic dependencies
9$ ldd main                     # what the loader will actually map
10$ addr2line -e main 0x1149     # an address back to file:line

Four commands produce four files; five commands read them. Nothing in this chain is opaque, and the habit of stopping one stage before the failure and looking at what came out resolves most build problems faster than reasoning about them does.

The build system is not part of the toolchain, and treating it as one causes trouble

typicalThis describes the conventional Unix-style toolchain that GCC and Clang present, which covers Linux, macOS and the BSDs. MSVC divides the work differently — cl.exe and link.exe, a different object format, a different debug-information format (PDB rather than DWARF), and a different mapping from error text to responsible program. Cross-compilation adds another axis, since the toolchain that runs and the toolchain being targeted are different sets of these programs — see [[cross-compilation]].

The five programs above are a fixed pipeline: given the same inputs they do the same thing. The build system is a different kind of object — it decides *which* invocations happen at all, based on a dependency graph and a notion of what is out of date. Its failure modes are therefore about staleness and correctness of that graph rather than about any program's input.

The classic failure is an undeclared dependency: a rule reads a file it never declared, so a change to that file does not trigger a rebuild, and the build produces a stale artifact that is correct for yesterday's source. Nothing in the compiler, assembler or linker is wrong; the graph was incomplete. This is why generated headers and code-generation steps cause so much trouble, and why [[build-dependency-graph]] and [[hermetic-compilation]] are lessons rather than footnotes.

The related confusion is between the driver and the build system. gcc and clang are drivers: they run several of these programs in sequence for one translation unit. They are not build systems and know nothing about your project — no dependency tracking, no parallelism policy, no notion of what is out of date. Conflating the two produces builds that "work" until they are run incrementally.

How it works

The steps, in the order the compiler takes them.

  • The driver reads the command line and decides which of the underlying programs to run, in what order, with what flags.
  • The preprocessor produces one self-contained translation unit; the compiler turns it into assembly or object code; the assembler encodes instructions and records symbols and relocations.
  • The linker resolves undefined symbols against other objects and libraries in command-line order, applies relocations, merges sections and writes an image with program headers.
  • At run time the loader maps the image and its dynamic dependencies into a process address space and binds their symbols — lazily for functions, unless configured otherwise.
  • Debug information emitted alongside the code maps machine addresses back to source lines and variable locations, which is what a debugger and a symbolizer read.
  • Above all of this, a build system maintains a dependency graph, decides which outputs are out of date, and issues the invocations in an order the graph permits.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • A linker error is treated as a compiler error, and hours are spent editing source for a problem that was a missing library on the command line.
  • A build succeeds and the program fails to start on another machine because a shared library is absent or the wrong version — a failure no build stage could have predicted.
  • A stale artifact is produced after a change to a file the build system did not know was an input, and the resulting behaviour matches neither the old source nor the new.
  • Two libraries define the same symbol, the linker silently takes the first in command-line order, and the program's behaviour depends on link order rather than on any source file.
  • A debugger reports "no debugging symbols found" because the build stripped them, or a variable as optimized out because the optimizer removed it — two different problems with similar-looking symptoms.
  • A cross-compilation build accidentally invokes the host assembler or linker, and the error names an instruction or an object format that has nothing to do with the target.

When it helps

  • Diagnosing any build failure, where the first useful question is always which program produced the message.
  • Substituting a component — a faster linker, a different assembler, a compile cache — which is only possible because the intermediate formats are documented.
  • Setting up cross-compilation, where every one of these programs must be the target's rather than the host's.
  • Understanding why a program that built fine fails at startup, which is a loader question and unaffected by anything in the source.

When it hurts

  • Treating the driver as the whole story. gcc hides the pipeline by design, which is convenient until the hidden stage is the one failing.
  • Assuming the same error text on another platform. MSVC divides the work differently and phrases failures differently, so the mapping from message to program has to be relearned.
  • Expecting the build system to catch a toolchain problem, or the toolchain to catch a dependency-graph problem. Neither can see the other's domain.

What it costs

Every one of these is paid by something.

  • Separate programs with documented formats buy inspectability and the ability to replace any stage independently, and pay with a chain of artifacts, more processes, and a diagnosis skill that has to be learned.
  • The driver hiding the pipeline buys usability for the common case, and pays by making the pipeline invisible exactly when it fails.
  • Emitting debug information buys debugging and symbolication, and pays with much larger object files and binaries — often the dominant term in build output size.
  • A build system that tracks dependencies precisely buys correct incremental builds, and pays with the effort of declaring every input, including generated ones, correctly and forever.

What else you could do

What a different compiler or language does instead, and when that is better.

  • An integrated compiler that emits object code directly, skipping the assembly text stage, is faster and gives up the readable intermediate — most compilers do this by default and re-enable it with -S.
  • Whole-program tools that fuse compilation and linking, as [[link-time-optimization]] does, buy cross-unit optimization at the cost of a much slower and much more memory-hungry link.
  • Single-binary toolchains such as Go's or Zig's present one command for everything, which is simpler to use and less separable when a stage misbehaves.
  • Statically linking everything removes the loader's role in finding libraries, trading deployment simplicity for binary size and rebuild-everything patching — see [[static-linking]].
  • Interpreted and JIT-based runtimes collapse most of this chain into one process, which is why their failure vocabulary is completely different — see [[python-pipeline]].

See it for yourself

The flag, dump or tool that shows you this directly.

  • gcc -### prints the exact sub-commands the driver would run, with all implicit flags, without running them — the single best way to see the hidden pipeline.
  • gcc -E, -S, -c stop after the preprocessor, compiler and assembler; -v prints each stage as it runs.
  • nm -C, objdump -d, readelf -a and otool inspect object files and executables; strings and size answer cruder questions quickly.
  • ldd, otool -L, LD_DEBUG=libs and DYLD_PRINT_LIBRARIES=1 show what the loader will actually map and in what order it searched.
  • addr2line -e binary <addr> and llvm-symbolizer turn an address from a crash report back into a source location, if debug information survived.
  • ld -Map=out.map (or -Wl,-Map=) produces a linker map showing which object supplied each surviving symbol — the answer to most "which copy won" questions.

Plausible wrong readings

Stated the way a confident engineer states them.

  • "The compiler said undefined reference." The linker did. The compiler was satisfied; it was told the symbol exists elsewhere and believed it.
  • "gcc is the compiler." gcc is a driver that runs a preprocessor, a compiler, an assembler and a linker. -### shows all four.
  • "If it builds, it will run." Building says nothing about whether the loader can find the shared libraries the executable records.
  • "The build system is part of the toolchain." It decides which toolchain invocations happen. Its failures are about dependency tracking and are invisible to every program it invokes.
  • "Debug information is only for debugging." Symbolication of production crash reports, profilers and sanitizers all read it, which is why stripping it has consequences beyond interactive debugging.

Misconceptions

The claim, and what is actually true.

The compiler produces the executable.
It produces object code for one translation unit. The linker produces the executable, and it is the only stage that sees the whole program.
A successful build means every symbol was resolved.
Every symbol needed at link time was resolved. Dynamically linked symbols are bound by the loader, potentially lazily, and can fail long after the build.
Stripping a binary only affects debugging.
It affects crash symbolication, profiling and anything that needs to map an address to a name. The usual answer is to strip the shipped binary and keep the symbols separately.
The build system guarantees a correct incremental build.
It guarantees a build consistent with the dependency graph it was given. An undeclared input produces a stale artifact with no error anywhere.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

Building a program runs several separate tools in a row: one pastes headers together, one compiles, one encodes instructions, one combines everything into an executable. Later, when you run it, another one loads it into memory. Each has its own error messages, and knowing which tool is talking tells you what kind of fix to look for.

practical

Two commands earn their keep. gcc -### shows the whole hidden pipeline with every implicit flag, which resolves most "why is it using that library" questions instantly. And stopping one stage before the failure — -E, -S, -c — and looking at the artifact beats reasoning about what should have happened. For link problems specifically, -Wl,-Map=out.map tells you which object actually supplied each symbol, which is the question you usually have.

advanced

The design point worth extracting is that the separations are defined by *file formats*, not by function boundaries, and that is what makes them durable. Because an object file is a documented format rather than an internal handoff, a linker written thirty years later can be dropped into a build unchanged, a compile cache can memoise the compiler by hashing its inputs, and a distributed build can ship translation units to other machines. Every one of those tools exists because the boundary was a format. The counter-pressure is equally real: link-time optimization deliberately breaks the boundary by putting IR in object files, so the "linker" now runs a code generator and the tools that read object files stop working as expected. That tension — inspectable boundaries versus cross-boundary optimization — recurs everywhere in this domain, and it is the same trade [[separate-compilation]] and [[whole-program-optimization]] describe from the other side.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

typicalThe six-program division describes conventional Unix-style toolchains as presented by GCC and Clang. MSVC splits the work between cl.exe and link.exe, uses COFF objects and PDB debug information rather than ELF and DWARF, and phrases its failures differently, so the mapping from message to responsible program must be relearned per platform.
implementationMost compilers emit object code directly rather than producing assembly text and invoking a separate assembler; -S re-enables the textual stage for inspection. Similarly, link-time optimization changes what is in an object file — IR rather than machine code — so objdump -d on an LTO object shows almost nothing, which surprises people who have not met it before.
targetObject formats, debug-information formats, symbol naming and dynamic-loader search rules are all per-platform: ELF with DWARF on Linux, Mach-O with DWARF and dSYM bundles on macOS, PE/COFF with PDB on Windows. The tools listed here are the ELF ones; the macOS and Windows equivalents differ in name and in output.

If you were asked this in an interview

  • A build fails with "undefined reference". Which program produced that, and what classes of cause does it imply?
  • Walk me through every program that runs between a .c file and a running process.
  • Why can a program build successfully and still fail to start on another machine?

Connections

Securitysupply-chain-security
OS & Networkingprogram-vs-processmemory-mapping
Domains that do not exist yet
  • DevOps / Production Engineering — Build systems, artifact caching and reproducible builds as infrastructure
    The build system is the one component here that is not a compiler program at all — its correctness is about dependency graphs, caching and hermeticity, which are build-engineering concerns. This lesson owns only the boundary where it invokes the toolchain.
  • Operating Systems — The dynamic loader, address-space layout and symbol binding at process start
    The loader is the only stage in this chain that runs inside the operating system rather than as part of a build, and how it maps segments and binds symbols is owned there. We own what the linker handed it.