Infraimplementation

Clang

Clang is a C, C++ and Objective-C frontend that lowers to LLVM IR — and, unusually, a library whose AST is a supported product in its own right. That second decision is why clang-format, clang-tidy and clangd exist and why they agree with the compiler.

The question

What does Clang do that LLVM does not, and why is so much C++ tooling built on it?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute
What the program is here

A translation unit as a Clang AST: a tree that keeps source locations for everything, retains constructs a code generator does not need — implicit conversions as explicit nodes, macro expansion locations, the sugar of a typedef alongside the type it names — and can answer questions about the program as written rather than about the program as compiled. That extra fidelity is the entire product. After it, the same frontend lowers to LLVM IR and the tree is no longer needed, but by then a dozen tools have already read it.

What this phase may assume or do

The frontend may lower to IR only what the language standard defines, and it must encode as attributes and flags every assumption the standard licenses but the IR cannot infer — signed overflow being undefined, restrict meaning noalias, a function being noexcept. The tooling half carries a different obligation: a tool operating on the AST may assume the tree is exactly what the compiler saw for that translation unit, which holds only if it was given the same compilation command. That is what compile_commands.json is for, and a tool run without it is analysing a different program from the one that gets built.

Key points

  • Clang is a C-family frontend that lowers to LLVM IR; everything after that seam is shared with every other LLVM client.
  • Its command-line driver is deliberately GCC-compatible, which is what allowed it to be adopted into existing build systems.
  • Its AST is a supported library product, which is why the C++ tooling ecosystem is built on the compiler's own representation.
  • That AST is high-fidelity on purpose: source ranges, implicit nodes, typedef sugar and macro provenance are all retained.
  • Tools need the compilation command to see the same translation unit as the build, which is what a compilation database provides.
  • Diagnostic quality was a design goal paid for with memory per node and frontend complexity, not a matter of effort.

A frontend, a driver, and a library — three products in one binary

implementationClang's AST is a supported product but not a stable ABI: the C++ classes change between releases like the rest of LLVM, which is why LibClang exists as a narrower, more stable C interface for tools that cannot track that cadence. Tools written against the C++ API are pinned to a version; tools written against LibClang give up detail in exchange for surviving upgrades.

The compiler proper is the part that turns C-family source into LLVM IR: preprocessing, lexing, parsing, semantic analysis, template instantiation, constant evaluation, and lowering. Everything downstream of that seam is [[llvm-architecture]], shared with every other frontend.

The driver is a separate concern that gets less attention than it deserves. clang the command is GCC-command-line-compatible by design: it accepts the same flags, runs the same conceptual sequence of preprocessor, compiler, assembler and linker, and can be dropped into a build system that has only ever seen gcc. That compatibility is a deliberate adoption strategy and a substantial engineering commitment on its own.

The third product is the one that changed the ecosystem. Clang's AST and its supporting machinery are exposed as libraries — LibTooling for whole-tool authoring, LibClang as a stable C interface, and AST matchers as a declarative query language over the tree. The tree is a supported thing to build on rather than an internal detail that happens to be reachable. That is the concrete form of [[ast-as-shared-infrastructure]].

Clang, and where it hands overimplementation
  1. Source + headersyou write it
    A .c, .cpp or .m file and everything it includes.
  2. Preprocessed translation unitbuild time
    One self-contained text with includes pasted in and macros expanded.
    Self-containment; and a record of where each expansion came from, which is why Clang can point diagnostics at a macro definition and its use site.
    Nothing, unusually — the expansion locations are retained rather than discarded.
  3. Clang ASTbuild time
    A high-fidelity tree with source locations, implicit nodes, sugar and macro provenance.
    A representation faithful enough to answer questions about the code as written — which is what makes a formatter and a refactoring tool possible.
  4. Semantic analysisbuild time
    The same tree with types, overload resolution, template instantiation and constant evaluation complete.
    Every diagnostic the compiler produces, and the proof that the unit is well-formed.
  5. CodeGen to LLVM IRbuild time
    An LLVM module with attributes encoding what C or C++ licensed.
    The seam. Everything about C++ stops here and everything about optimization begins.
    The AST, the sugar, the macro provenance — kept only as far as debug metadata carries it.
  6. LLVM middle-end and backendbuild time
    Optimized IR, then machine code.
    Optimization and code generation shared with every other LLVM client.

Read it asOnly the last row is not Clang, and only the first four are visible to tooling. The loses row at the CodeGen seam is why a refactoring tool must run before it: implicit conversions, typedef sugar and macro provenance are exactly the things you need to rewrite source faithfully, and exactly the things a code generator has no use for.

Why the tooling followed

Before Clang, a C++ tool that needed to understand code had two options: reimplement a C++ parser, or parse approximately with regular expressions and hope. The first is a multi-year project that will still disagree with the compiler; the second produces tools that break on templates, macros and anything unusual. The result was an ecosystem of tools that all understood a slightly different C++ from the one being compiled.

Making the compiler's own frontend a library removed that whole category of problem. clang-format reformats using the real token stream and the real understanding of what is a template argument list rather than a comparison. clang-tidy checks and rewrites using AST matchers over the real tree, so a check can say "this is a copy of a std::vector in a range-for" with certainty rather than heuristically. clangd provides completion, diagnostics and go-to-definition in an editor by running the actual frontend — see [[language-server]] and [[lsp]]. The sanitizers are the same idea on the code-generation side: instrumentation inserted by the compiler that knows exactly what the program means.

The precondition for all of it is that these tools need the compilation command. C++ has no notion of a project; a translation unit is only defined by the flags and include paths it was compiled with. compile_commands.json — a compilation database that build systems emit — is how a tool learns them, and a tool run without one is analysing a program that differs from the one being built in exactly the ways that matter.

What each tool reads, and what that lets it be certain aboutimplementation
ToolReadsCertain about
clang-formatThe token stream, with enough parsing to disambiguateWhat is a template argument list rather than a comparison; where a statement ends
clang-tidyThe full AST, via matchersTypes, overloads, implicit conversions — so a check can rewrite code safely
clangdThe AST, incrementally, plus a project indexEverything the compiler knows, which is why its diagnostics match the build
Sanitizers (ASan, UBSan, TSan)The IR at code generation, with instrumentation insertedWhich accesses need a check and what the source-level operation was
Clang Static AnalyzerThe AST plus a path-sensitive exploration of the CFGFacts along specific paths — at a cost in analysis time — see [[static-analysis]]

Diagnostics as a design goal, not a polish item

Clang's early reputation was built on error messages, and it is worth being precise about why they were better rather than treating it as a matter of effort. The messages are good because the representation supports them: source ranges on every node rather than a single position, macro expansion backtraces, retained typedef sugar so a message can say std::string instead of the underlying template instantiation, and enough recovery to keep parsing and report several real errors instead of one and a cascade.

None of that is free. Retaining sugar, ranges and provenance costs memory on every node and discipline in every transformation, and error recovery costs a great deal of frontend complexity for behaviour that only matters on invalid programs. It is a deliberate purchase, and the general argument is [[diagnostic-quality]] and [[error-recovery]].

The competitive effect is the interesting part of the history. GCC's diagnostics improved substantially in the years after Clang appeared — column numbers, caret positions, fix-it hints, better template error output. That is not a story about one project being better than another; it is a story about a design decision propagating because it turned out to be worth its cost. Both toolchains now spend real effort here.

How it works

The steps, in the order the compiler takes them.

  • The driver parses a GCC-compatible command line and decides which stages to run and with what flags.
  • The preprocessor expands includes and macros while recording the provenance of every expansion.
  • A hand-written recursive-descent parser builds the AST, attaching a source range to every node and inserting explicit nodes for implicit conversions.
  • Semantic analysis resolves names and overloads, instantiates templates, evaluates constant expressions and produces diagnostics with ranges and fix-it hints.
  • Tools may attach here: AST matchers query the tree, and LibTooling runs a tool over one translation unit given its compilation command.
  • CodeGen lowers the checked AST to an LLVM module, emitting attributes for every assumption the language licenses and debug metadata for the source correspondence.
  • LLVM optimizes and generates code; the driver then invokes an assembler and a linker as required.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • A clang-tidy run reports errors that the build does not, because it was run without a compilation database and guessed include paths that do not match.
  • A tool built against the C++ AST API stops compiling after an LLVM upgrade, because that API is unstable by policy.
  • An editor's diagnostics disagree with the build for a file that is compiled with different flags in different targets — the same source, two translation units, two answers.
  • A refactoring tool rewrites code correctly for the instantiation it saw and wrongly for another, because a template body is only checked against the arguments actually used.
  • A sanitizer-instrumented build is several times slower and much larger, and someone deploys it by accident because the flag was set in a shared configuration.
  • A fix-it hint is applied automatically at scale and is right in every case except the ones inside macros, where the rewrite lands in the definition rather than the use.

When it helps

  • Building any tool that needs to understand C or C++ accurately, where reimplementing the frontend is not a viable plan.
  • Getting editor diagnostics that match the build exactly, which removes an entire class of "works for me" confusion.
  • Large-scale automated refactoring, where AST matchers plus fix-it hints can apply a change across a codebase with the compiler's own understanding of what the code means.
  • Debugging memory and undefined-behaviour bugs with sanitizers, which know the source-level meaning of each access.

When it hurts

  • Building a tool that must survive LLVM upgrades without maintenance. The C++ API changes every release, and LibClang trades detail for stability.
  • Analysing code without the real build flags. A translation unit is defined by its command line, and a tool without one is analysing something else.
  • Assuming a Clang-specific extension or attribute is portable. Much of what it accepts is shared with GCC, and some is not.

What it costs

Every one of these is paid by something.

  • A high-fidelity AST buys accurate tooling and excellent diagnostics, and pays with memory per node and the discipline of preserving ranges and sugar through every transformation.
  • Exposing the AST as a product buys an ecosystem, and pays with a compatibility obligation — every tool now depends on the tree shape, so changing the frontend means coordinating with tools nobody in the project controls.
  • GCC command-line compatibility buys drop-in adoption in existing builds, and pays with a large surface of legacy flag behaviour that must be reproduced and maintained.
  • Error recovery buys several real diagnostics per run instead of one and a cascade, and pays with frontend complexity spent entirely on programs that will not compile.

What else you could do

What a different compiler or language does instead, and when that is better.

  • GCC is a complete alternative toolchain with a different internal architecture and a plugin-based extension story rather than a library-based one — see [[gcc]].
  • MSVC is the third mature C++ implementation, with its own frontend, object format and mangling scheme, and its own tooling story.
  • EDG is a commercial C++ frontend licensed by other vendors, which is how several toolchains obtain a conforming parser without writing one.
  • A regex- or heuristic-based tool needs no compiler infrastructure and is wrong on templates, macros and anything unusual — the situation Clang's libraries removed.
  • Tree-sitter parses fast and incrementally without types, which is the right choice for syntax highlighting and the wrong one for anything semantic.

See it for yourself

The flag, dump or tool that shows you this directly.

  • clang -Xclang -ast-dump -fsyntax-only file.cpp prints the AST, including implicit nodes; -ast-dump=json for machine consumption.
  • clang -Xclang -dump-tokens shows the token stream after preprocessing; clang -E shows the preprocessed text itself.
  • clang-query gives an interactive REPL for AST matchers, which is how you develop a clang-tidy check without recompiling anything.
  • clang -S -emit-llvm -o - shows exactly what the frontend hands to LLVM, including the attributes it chose to emit.
  • -MJ per file, or CMake's CMAKE_EXPORT_COMPILE_COMMANDS=ON, produces the compile_commands.json every tool needs.
  • clang -fsanitize=address,undefined for the instrumentation path; -ftime-trace for where frontend time went.

Plausible wrong readings

Stated the way a confident engineer states them.

  • "Clang and LLVM are the same project, so the names are interchangeable." Clang is one frontend; LLVM is the infrastructure it and a dozen other frontends emit into.
  • "Clang has better error messages because its developers cared more." It has better error messages because the AST retains ranges, sugar and macro provenance, which costs memory and complexity. The design paid for the messages.
  • "clang-tidy is a linter, so it works like ESLint on any file." It needs the real compilation command, because a C++ translation unit does not exist without one.
  • "If clangd agrees, the build will agree." Only if clangd used the same flags. Different targets compiling the same file differently is the normal case in large builds.
  • "The AST is stable, so my tool will keep working." The AST is a supported product with an unstable C++ API. LibClang is the stable interface, and it deliberately exposes less.

Misconceptions

The claim, and what is actually true.

Clang replaced GCC as the standard C++ compiler.
Both are mature, actively developed and default on different platforms. Which one is the default depends on the operating system, the distribution and the project.
The Clang AST is what gets compiled.
It is lowered to LLVM IR and then discarded. The AST is what tools read; the IR is what becomes code, and the two contain deliberately different information.
Tooling built on Clang understands my project.
It understands the translation units you give it, with the flags you give it. Without a compilation database it is guessing at both.
Sanitizers are a kind of static analysis.
They are compiler-inserted runtime instrumentation. They find bugs on paths that actually execute, which is a different and complementary capability to analysing paths statically.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

Clang is the part that understands C and C++. It reads your code, checks it, produces error messages, and hands a neutral description of the program to LLVM, which does the optimizing and produces machine code. Its unusual feature is that the tree it builds while understanding your code is available to other programs, which is why the formatter, the linter and the editor plugin all agree with the compiler.

practical

Two habits. First, make sure everything has a compilation database — CMAKE_EXPORT_COMPILE_COMMANDS=ON or the equivalent — because without it every tool is analysing a different program from your build, and the resulting confusion is expensive and hard to attribute. Second, when you want to know what the compiler actually thinks a construct means, clang -Xclang -ast-dump -fsyntax-only answers it in seconds, and clang-query lets you explore the tree interactively before writing a check.

advanced

The decision worth studying is the one to make the AST a product. Most compilers keep their tree private, and for good reasons: it lets the frontend evolve freely, and it avoids committing to shapes that were internal conveniences. Clang gave that up and got an ecosystem in return, and the cost is visible in the project — AST changes now have downstream consumers, LibClang exists as a stability compromise, and matcher-based checks constrain how nodes may be reorganised. It is worth comparing with Python, where the ast module has been public for far longer and the same dynamic produced the same ecosystem and the same constraint, and with JavaScript, where no engine exposes its tree and the ecosystem responded by building half a dozen independent parsers that each disagree slightly. Three languages, three positions, and the tooling landscape of each follows directly from the choice — see [[ast-as-shared-infrastructure]].

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

implementationClang's C++ AST API changes every LLVM release by the same policy that governs the rest of the project, so tools written against it are pinned to a version; LibClang is the narrower C interface for tools that must survive upgrades. Which language extensions and standard-library features are supported also differs by release and by the standard library actually in use — libc++, libstdc++ or the MSVC STL.
specWhat Clang must accept and reject is specified by the C and C++ standards, so conforming code behaves the same on Clang, GCC and MSVC. What is not specified is anything about the AST, the diagnostics, the extension set or the tooling, none of which any standard mentions — a program that relies on those relies on one implementation.
typicalThe claim that Clang and GCC produce comparable code quality is the general pattern across mainstream targets and workloads rather than a universal result: each wins on particular programs and particular architectures, and vendor toolchains beat both on their own hardware. Any comparison is a measurement of specific code with specific flags, not a ranking of the projects.

If you were asked this in an interview

  • Where does Clang stop and LLVM start, and what crosses the boundary?
  • Why does clang-tidy need a compilation database when a JavaScript linter needs nothing?
  • What in Clang's AST design makes its diagnostics good, and what does that cost?

Connections

Performancebenchmarking
Domains that do not exist yet
  • Developer Tooling and Editor Integration — The editing experience a language server is trying to deliver
    Everything about latency budgets, incremental reparsing under an editor's keystroke rate and what a developer perceives as responsive is a tooling and product concern. This lesson owns only the compiler-side fact that makes it possible: a frontend that can be run as a library.