Designspec

Syntax versus Semantics

Syntax decides which strings are programs; semantics decides what they mean. The reason to keep them apart is that a compiler phase can only enforce one of them, and almost every argument about a language is an argument about the wrong one.

The question

What is actually the difference between a syntax error and a type error, and why do languages draw the line differently?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute
What the program is here

Two separate artifacts in the definition. The syntax is a grammar over tokens, and the only question it answers is whether a string is a well-formed program. The semantics is a pair of relations over well-formed programs — a static semantics saying which of them are meaningful, and a dynamic semantics saying what a meaningful one does. The parser produces a tree from the first and knows nothing about the second, which is exactly why the tree can be well-formed and meaningless.

What this phase may assume or do

The parser may reject only what the grammar forbids, and must accept everything the grammar admits, including programs that are obviously nonsense — "hello" + 3 is syntactically flawless in most languages, and rejecting it there would put a type rule in the parser where no later phase can find it. Symmetrically, the type checker may assume the tree is well-formed and may not re-derive structure. Violating either direction produces a compiler whose errors arrive from the wrong phase with the wrong information.

Key points

  • Syntax decides which strings are programs; static semantics decides which programs are meaningful; dynamic semantics decides what they do.
  • The same meaning can wear three different syntaxes with no change to anything after the parser.
  • The same syntax can carry four different meanings, and in one of them assignment moves rather than copies, which changes what code after it is legal.
  • Where a language draws the line is a design decision with permanent tooling consequences — C's type-dependent grammar being the most expensive example.
  • Diagnostic quality follows from the split: a parser can say what it expected, a checker can say what you meant, and neither can do the other's job.

Same meaning, different syntax

The cleanest way to see that these are independent is to hold one fixed and vary the other. Below is one semantics — bind a name to the result of adding two numbers, then return it — written three ways. Nothing about the compiler after the parser differs. The AST is identical, the type rules are identical, the generated code is identical.

This is why syntax arguments are cheap in the technical sense and expensive in the social one: changing the surface costs a parser change and nothing downstream, and it changes how every program in the language reads forever. Both halves of that are true simultaneously, which is why the arguments never resolve.

Three grammars, one language
C-family::=let = "int" IDENT "=" expr ";"Type first, explicit terminator, braces for blocks.
ML-family::=let = "let" IDENT "=" expr "in" exprBinding is an expression, so it has a body rather than a scope delimited by punctuation.
Layout-sensitive::=let = IDENT "=" expr NEWLINEThe lexer emits indent and dedent tokens, so the grammar can stay context-free while the source has no braces — `[[lexical-analysis]]`.
Deriving x = a + b
  1. 1.letapplying start
  2. 2.IDENT "=" exprapplying let production
  3. 3.IDENT "=" expr "+" exprapplying expr -> expr "+" expr
  4. 4.x = a + bapplying terminals

Same syntax, different meaning

specEach row is a language-definition fact rather than a compiler behavior, which is why the differences are stable across implementations. The Rust row in particular is a definition-level consequence of move semantics: any Rust compiler must reject the second use, and the error is a borrow-check error rather than a type error, which is a third category the other three languages do not have.

The other direction is more consequential and much less discussed. Identical text, identical parse tree, and four different behaviors — because the semantics of assignment, of + and of the values involved differ per language.

Notice what this does to the compiler. In the first two rows the meaning is fixed at build time by the static types, and a type error is available. In the third it is fixed at run time by the values, and no static phase can decide it. In the fourth, the meaning of assignment itself differs: the same tree either copies, aliases, or moves, and the difference decides whether a later use of the source variable is legal.

b = a; c = a + a; — one parse tree, four semanticsspec
LanguageWhat `b = a` doesWhat `a + a` doesWhich phase decides
C, a an intCopies the valueInteger addition; signed overflow is undefinedType checker, from the declared type
C++, a a std::stringCopies, invoking the copy constructor and allocatingConcatenation, allocating a new stringOverload resolution, from the static type
Python, a unknownBinds the same object to a second name; nothing is copiedConcatenation for strings and lists, addition for numbers, TypeError for a mixNothing at build time. The run decides, per execution
Rust, a a StringMoves. a is no longer usable afterwardsRejected: String does not implement Add<String>The borrow checker and trait resolution, both at build time

Where the line goes is a design decision

It would be tidy if syntax handled shape and semantics handled meaning with no negotiation. In practice, languages move rules across the boundary deliberately, and each move has consequences for diagnostics and for tooling.

Python makes indentation syntactic. The consequence is that a mis-indented block is a parse error with a precise location rather than a subtly wrong program, which is a real gain — and that a code generator must track indentation, and that mixing tabs and spaces becomes a lexical hazard the language must legislate about.

Rust makes ownership *not* syntactic. let b = a; is grammatically identical whether a is copied or moved, and the difference is decided by whether the type implements Copy. That keeps the grammar small at the cost of a rule readers must know rather than see, which is a recurring criticism and a deliberate trade.

C makes a much stranger move: the grammar is not context-free, because A * B; is a declaration if A is a type name and a multiplication expression if it is not. The parser therefore needs the symbol table, and the two phases cannot be cleanly separated in any C or C++ implementation. That is one design decision, made early, that permanently shapes every tool anyone builds for those languages. See [[context-free-grammars]] and [[semantic-analysis]].

What each phase can and cannot say

The practical payoff of the distinction is diagnostic quality. A parser knows what token it expected and where; it does not know what you meant. A type checker knows what you meant and what would have made it work; it cannot tell you about a missing brace, because the parser never got that far.

This is why a single missing brace produces forty errors in some languages and one in others. The parser lost synchronisation, kept going, and every subsequent construct was interpreted in the wrong context. Recovering well is a real engineering problem with real techniques — [[error-recovery]] and [[parser-synchronization]] — and it is entirely a syntax-side problem that no amount of semantic analysis can help with.

A syntactically perfect, semantically meaningless program
AST — only what later phases match on
Assign“n = "hello" + 3”
├── Identifier n“n”— The parser does not know whether `n` was declared. It is not allowed to care.
└── Binary +“"hello" + 3”
├── StringLiteral "hello"“"hello"”
└── IntLiteral 3“3”

Read it asEvery grammar rule was satisfied, so the parser is finished and successful. Whether adding a string to an integer means anything is a semantic question with four different answers across mainstream languages — concatenation after coercion, a type error at build time, a type error at run time, or a compile error about a missing trait implementation. The tree is the same in all four.

How it works

The steps, in the order the compiler takes them.

  • The grammar defines the set of well-formed token sequences, and the parser decides membership and produces a tree.
  • The static semantics defines which well-formed trees are meaningful, expressed as typing and binding rules over the tree, and the checker applies them.
  • The dynamic semantics defines what a meaningful program does, and the rest of the pipeline is an implementation of it.
  • A rule can be moved between these layers deliberately: making indentation syntactic moves a class of mistake from the semantic layer to the parser, which changes where the error is reported and how precisely.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • One missing brace produces forty diagnostics, none of which mentions a brace, because the parser lost synchronisation and reinterpreted the rest of the file.
  • A type error is reported at a location far from the mistake, because inference propagated a constraint from a distant expression before finding the contradiction — see [[type-inference]].
  • A refactoring tool corrupts a file because it worked from an AST and the AST discarded the punctuation and comments that were needed to write the file back out.
  • A program that reads correctly to a human is rejected, because the grammar is ambiguous where the human is not and the language resolved the ambiguity the other way — the dangling else being the classic case.
  • A parser generator reports a shift/reduce conflict, and the developer suppresses it with a precedence declaration that silently changes what a whole class of programs means.

When it helps

  • Debugging a compiler error you do not understand: knowing which phase produced it tells you what information it had and therefore what it could not have known.
  • Designing diagnostics. The best error messages come from phases that were given enough context to say what was expected, which is a decision made when the phase boundary is drawn.
  • Evaluating a proposed syntax change. If nothing downstream of the parser changes, the argument is about readability and cost of churn, and saying so ends most of it.

When it hurts

  • Insisting on a clean separation where the language does not have one. C and C++ parsers need semantic information, and pretending otherwise produces a parser that is wrong on real code.
  • Treating syntax as unimportant because it is downstream-neutral. It is the entire interface to the language for every human who reads it, and readability under maintenance is a real design goal — [[language-ergonomics]].

What it costs

Every one of these is paid by something.

  • Pushing a rule into the grammar buys a precise, early error with an exact location, and costs grammar complexity and flexibility — significant indentation makes mis-nesting a parse error and makes generated code, minification and one-liners harder.
  • Leaving a rule to the semantics buys a small, context-free grammar that is easy to tool for, and costs error locality: a semantic error can be reported far from its cause because the phase that found it is working with a tree, not with the text.
  • A grammar that depends on semantic information buys expressive declaration syntax and costs the ability to parse the language independently — every tool, forever, needs a full frontend rather than a parser.

What else you could do

What a different compiler or language does instead, and when that is better.

  • S-expressions: make the syntax so uniform that parsing is trivial and all structure is explicit, which is what Lisp does. It buys macro systems that operate on the real structure and costs a surface that many readers reject outright.
  • A concrete syntax tree preserving every token, space and comment, so that formatters and refactoring tools can round-trip. Costs memory and traversal complexity in every consumer — [[concrete-syntax-tree]].
  • No syntax of your own at all: express the domain as types and functions in a host language, and inherit its grammar, its parser and its editor support — [[internal-vs-external-dsl]].

See it for yourself

The flag, dump or tool that shows you this directly.

  • The grammar-only view: python -c "import ast; print(ast.dump(ast.parse(open('f.py').read())))" shows exactly what the parser produced, before anything semantic ran.
  • The semantic view of the same file: mypy --strict f.py, or clang -Xclang -ast-dump which annotates the tree with resolved types.
  • The dividing line in C++: try clang -fsyntax-only on a file that uses an undeclared type name in a declaration. The error is not a parse error, which is the point.
  • Ambiguity in a grammar you wrote: run it through a parser generator and read the conflicts. A shift/reduce conflict is the grammar telling you two readings exist — [[ambiguous-grammars]].

Plausible wrong readings

Stated the way a confident engineer states them.

  • "A syntax error means my code is wrong." It means your text is not a program. Code that is entirely wrong can be syntactically perfect, and usually is.
  • "Semantics is just types." Types are the largest part of most static semantics, and binding, initialisation, exhaustiveness, ownership and effect rules are all semantics too, each with its own phase.
  • "Two languages with similar syntax are similar languages." C++ and Java share a great deal of surface and differ on assignment, memory, dispatch and generics — that is, on everything the syntax does not say.
  • "The parser validates my program." The parser validates its shape. In a language with a type-dependent grammar it also needs the symbol table, and even then it is checking shape.

Misconceptions

The claim, and what is actually true.

A well-formed program is a valid program.
Well-formed means the grammar accepted it. "hello" + 3 and a call to an undeclared function are both well-formed in most languages and both meaningless.
Parsers can detect type errors if you make the grammar rich enough.
Types depend on declarations elsewhere in the program, which a context-free grammar cannot express. Languages that try end up with parsers that consult the symbol table, which is C++ and is not tidy.
Significant whitespace is a stylistic preference.
It moves a class of error from the semantic layer into the parser, which changes both the diagnostics and what code generation looks like. It is a real design decision with real consequences on both sides.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

Syntax says which sequences of characters count as programs. Semantics says what those programs mean. A program can be perfect syntax and total nonsense, which is why compilers have separate phases and why "syntax error" and "type error" arrive from different places with different amounts of information.

practical

When an error confuses you, identify the phase. A parse error means the text stopped being a program at that point, and the real mistake is usually slightly before the reported position. A semantic error means the shape was fine and the meaning was not, and the reported position is where the contradiction was found, which in an inferring language can be far from where it was introduced. Fix parse errors strictly top to bottom; fix semantic errors by looking at what the checker thought the types were, not at where it complained.

advanced

The line between the two layers is a design variable, and moving it is the most underrated technique in language design. Every rule you move into the grammar becomes an error with an exact location and a cheap check, at the cost of grammar complexity. Every rule you move out becomes flexible and tool-friendly and reports worse. The languages usually held up as having excellent diagnostics — Rust, Elm — did not achieve that by writing better message strings; they achieved it by keeping spans on everything, by recovering well in the parser, and by choosing a layer for each rule that had enough context to explain itself.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

specThe four-way meaning table is drawn from language definitions, not from compiler behavior, so it holds across implementations. What does vary is the diagnostic: the same rejected Rust program produces different message text between versions, and the same Python TypeError appears at a different point depending on when the expression is reached.
typicalThe claim that a syntax change costs only a parser change holds for surface-level changes. A change that alters what is expressible — adding a new binding form, or making an expression form into a statement form — is a semantics change wearing syntax clothing, and it costs everything downstream.

If you were asked this in an interview

  • Give me a program that is syntactically valid and semantically meaningless, and say which phase rejects it and why not the other one.
  • Why can a C++ parser not be separated from its semantic analysis?
  • A missing brace produces forty error messages. What went wrong, and what technique fixes it?

Connections

DSAbinary-treestack
Domains that do not exist yet
  • Programming Languages & Runtime Internals — Dynamic semantics realised as an interpreter loop over a running program
    A dynamic semantics is a specification; a runtime is an implementation of one. This domain owns the specification and the translation; the machinery that executes the result is theirs.