Internal versus External DSLs
An internal DSL is written in the host language and inherits its entire toolchain for free. An external one has its own syntax and its own parser, and must build every tool from scratch. The choice is almost entirely about who pays for the tooling.
Should my domain language be embedded in the host language or have its own syntax?
Internal: a host-language program whose evaluation produces an inspectable value — a query object, a rule tree, a builder result. The host's parser produced it, the host's type checker checked it, and the domain program exists only at run time. External: a text file in its own grammar, parsed by a parser you own into an AST you own, existing as an artifact before anything runs. The representational difference is when the program exists — run time versus build time — and that difference is what determines which analyses are available.
An internal DSL may only express what the host's grammar accepts, so every construct must be a legal host expression; a checker for it may only assume what the host type system enforces plus what the builder API can validate at construction. An external DSL may assume whatever its own grammar and checker enforce, but only if every producer of its programs goes through that parser — a program constructed by string concatenation elsewhere has bypassed the guarantee entirely, which is the injection failure mode.
Key points
- Internal DSLs are host programs that build inspectable values; external DSLs are text in their own grammar, parsed by a parser you own.
- Internal inherits parser, type checker, editor support, debugger, package manager and test framework — all of the tooling bill.
- External buys notation the domain already uses and an artifact that exists before execution, and owes every tool.
- The characteristic internal failure is notation pushed past what the host supports, producing incomprehensible type errors.
- The characteristic external failure is a language whose diagnostics never improve past "unexpected token".
- If the audience is not host-language programmers, external is the only option that addresses the problem.
- How good an internal DSL can be is a property of the host language, so advice that does not name the host is useless.
- A DSL embedded in host strings has both sets of costs and neither set of benefits.
The same domain, twice
Take validation rules. Internally, they are host-language calls that build a rule object: field("age").required().min(0).max(150). Externally, they are a file in a notation of your own: age: required int in 0..150. Both are programs. Both can be inspected before running. They differ in almost everything else.
The internal version was parsed by the host's parser, so it already has syntax highlighting, formatting, go-to-definition, autocomplete, a type checker and a debugger. None of that was work. The external version has a notation that fits the domain exactly and, on day one, none of those things.
That is the trade in one sentence, and everything else in this lesson is detail. Internal buys the toolchain and constrains the notation. External buys the notation and owes the toolchain.
1// Internal: a host program that builds a value.2let rules = schema(3 field("age").required().int().between(0, 150),4 field("email").required().matching(EMAIL),5 field("role").required().oneOf(["admin", "user"]),6)7// Autocomplete works. A typo in `betwene` is a compile error.8// A domain error reads: "expected Between<int>, found Between<string>".9 10// External: a file in its own grammar.11age : required int in 0..15012email : required string matching EMAIL13role : required enum { admin, user }14// Reads better, and every tool that reads it is one you wrote.15// A domain error can read: "age: 0..150 excludes the default value 200".Compare the two error messages, not the two syntaxes. The internal form gets a compile error for free and it is phrased in host-language types the user never wrote. The external form can say exactly the right thing — if someone builds the diagnostics system that says it.
What internal inherits, in detail
The inherited list is longer than people expect and it is the whole argument. The host's lexer and parser: no grammar to write, no ambiguity to resolve, no error recovery to implement. The host's type checker: if the builder API is typed well, a large class of domain errors becomes a compile error with no checker of your own. The host's editor support: completion, hover documentation, go-to-definition, rename and find-references all work on domain code because it is host code.
And the host's runtime tooling: a debugger that steps through rule construction, a profiler that attributes time correctly, a package manager that versions the DSL like any other dependency, and a test framework that tests domain programs as ordinary values. Each of those is months of work for an external language, and they arrive free.
There is a fourth category that matters more than it sounds: composition. An internal DSL program is a host value, so it can be built by a function, stored in a variable, parameterised, and combined with ordinary code. External DSL programs compose only through whatever module system the language grew, which is usually late, ad hoc and the source of most of its complexity.
| Capability | Internal DSL | External DSL |
|---|---|---|
| Parser and error recovery | Inherited from the host | You write it, including recovery — see [[error-recovery]] |
| Syntax highlighting | Works already | A grammar per editor, or a language server |
| Type checking of domain code | Whatever the builder API encodes in host types | A checker you write, able to say exactly the right thing |
| Error message vocabulary | Host types, often naming things the user never wrote | The domain's own words, if you build it |
| Completion and go-to-definition | Free | A language server — see [[language-server]] |
| Debugging | The host debugger, on host code | Nothing, until you design what debugging even means |
| Notation | Bounded by host syntax | Whatever the domain already uses |
| Usable by non-programmers | Rarely — it is still host code | Achievable, and often the reason to build it |
| Whole-program analysis | Hard: the program exists only at run time | Natural: the artifact exists before execution |
| Versioning and migration | The host package manager | A policy you design, plus tooling to migrate old programs |
What internal gives up
The notation is bounded by host syntax, and the workarounds for that boundary are where internal DSLs go wrong. Operator overloading, method chaining, macros and clever type tricks can push a host language a long way toward a domain notation, and every step degrades the error messages. A builder chain that is invalid in a particular order produces a type error naming an intermediate generic type with six parameters, and the user has to reverse-engineer the API design to read it. That is the characteristic internal-DSL failure: beautiful notation, incomprehensible diagnostics.
The second loss is analysis. An internal program exists at run time, so whole-program questions — is this rule set complete, do these two rules conflict, what does this configuration change compared to the last one — require running the construction code and then inspecting the result. That is usually possible and it is never as convenient as a file you can parse, diff and check in CI without executing anything.
The third is the audience. If the reason for the DSL was that domain experts who are not programmers must read and write these programs, an internal DSL does not solve the problem: it is still host code, in host files, requiring a host toolchain to run. This is the one criterion that settles the question on its own, and it is the reason SQL, HCL and shader languages are external.
The hybrid, and the one that is neither
The most common real answer is a hybrid, and it is often the right one. Start internal, because it costs a fraction as much and is a cheap experiment in notation; when the notation has stopped changing and the audience genuinely needs its own file format, extract an external syntax that compiles into the same internal representation. The internal version becomes the compilation target, the semantics are already tested, and the external language starts life with a working back end.
Another hybrid: an external file format for the declarative parts and host code for anything that needs abstraction, with a defined boundary between them. Most successful configuration systems end up here, whether or not they intended to.
And then there is the shape that is neither, which is where most teams accidentally arrive: a DSL embedded in *strings* inside the host language. SQL in a string, a query fragment concatenated from parts, a rule expression evaluated with eval. It has all the costs of an external DSL — no host checking, no highlighting, no completion — and none of its benefits, since it has no parser of its own either. It is also where injection vulnerabilities come from, because the composition mechanism is string concatenation. If a domain notation is going to live inside strings, it needs a parser and a checker exactly as an external language would; the fact that it is spelled inside quotation marks changes nothing about the obligation.
- Start internal to explore the notation cheaply; extract external once it has stabilised and the audience needs it.
- Keep the internal representation as the external language's compilation target, so the semantics are shared and tested once.
- Declarative parts external, abstraction in the host, with an explicit boundary — the shape most configuration systems converge on.
- A DSL living in host strings has the costs of both and the benefits of neither, and is where injection bugs live.
- If the audience is non-programmers, the question is settled: external, or no DSL at all.
How it works
The steps, in the order the compiler takes them.
- Internal: design an API whose calls construct a value rather than perform work, using the host's type system to make invalid combinations unrepresentable where possible.
- Internal: keep the constructed value inspectable, so it can be validated, optimized, printed and tested independently of executing it.
- External: write a grammar, a lexer and a parser producing an AST with source spans on every node, because every later diagnostic depends on those spans.
- External: build a checker that reports in the domain's vocabulary, and an error-recovery strategy so one mistake does not hide the rest.
- External: choose an execution strategy — interpret, compile to the host, compile to bytecode — see
[[dsl-implementation-strategies]]. - Hybrid: compile the external syntax into the internal representation, so both front ends share one semantics and one test suite.
- Either way: decide the versioning story before the first user, because the migration cost is proportional to the number of existing programs.
How it breaks
What the engineer observes when it goes wrong — not what goes wrong internally.
- A fluent internal API produces a type error naming a generic type with six parameters that the user never wrote, and they cannot tell which call was wrong.
- An internal builder has valid and invalid call orderings that are documented only in the examples, so users discover the constraint at run time.
- An external DSL ships with a parser and no error recovery, so a missing brace on line 3 produces forty errors and the real one is not among them.
- An external DSL has no editor support, so users edit it in a plain text buffer with no feedback until they run the build.
- A DSL embedded in host strings is composed by concatenation and a user-supplied value changes the meaning of the program — the injection failure.
- An external DSL grows a module system late, and the ad hoc design becomes the most complex and least understood part of the language.
- An internal DSL is chosen for a non-programmer audience, and the audience cannot use it because it still requires the host toolchain to run.
When it helps
- Internal: when the readers already work in the host language and the value is naming the domain concepts rather than changing the syntax.
- Internal: when the domain is still moving, since changing an API is a deprecation and changing a syntax is a migration.
- External: when the audience is not host-language programmers and needs to read and write programs without a host toolchain.
- External: when whole-program analysis matters — checking, diffing or planning the program in CI without executing it.
- External: when the domain has an established notation that host syntax genuinely fights.
When it hurts
- Internal, when the host cannot carry the notation and the workarounds destroy the error messages.
- Internal, when the program needs to be analysed without running it, which requires executing the construction code first.
- External, when nobody is funded to build and maintain the tooling, which caps adoption at the author.
- External, when the domain is still changing, since every change is now a syntax migration for every existing program.
- Either, when the actual requirement was a well-named library and the DSL is a solution looking for one.
What it costs
Every one of these is paid by something.
- Internal buys the host's entire toolchain at no cost and pays in notation: the syntax is bounded by the host, and pushing past that boundary buys prettiness with diagnostics.
- External buys the domain's own notation and a build-time artifact, and pays for every tool separately — parser, recovery, formatter, language server, debugger, migration tooling — indefinitely.
- Internal buys cheap change, since evolving an API is a deprecation, and pays in analysability: the program exists only after the construction code has run.
- The hybrid buys a cheap notation experiment before committing to a syntax, and pays by maintaining two front ends over one semantics, with the internal API now a public interface it may not have been designed to be.
What else you could do
What a different compiler or language does instead, and when that is better.
- A plain library with excellent naming, which gets much of the readability with none of the design burden and is the right answer more often than either option here.
- A data format with a schema, which is external in form and needs no parser of your own — the schema tooling is the checker — see
[[configuration-languages]]. - Code generation: write a specification, generate host code from it, and let the host toolchain debug the output. The specification is the DSL and the generated code is ordinary.
- Host macros where the host has them, which is internal in cost and external in notation — the best of both, available only in Lisp-family languages, Rust, Scala and a handful of others.
See it for yourself
The flag, dump or tool that shows you this directly.
- For an internal DSL: print the value your builder produces before executing it. If it is an inspectable tree, you have a program; if the work already happened, you have a library.
- Introduce a deliberate error in an internal DSL and read the message as a new user would. If it names types the user never wrote, that is the tooling cost arriving.
- For an external DSL, delete a closing brace in the middle of a file and count the errors reported. One is good design; forty means no error recovery.
- Compare
psqlrunning a query string against a query built by an ORM:EXPLAINshows the same plan, which demonstrates that the internal and external forms are the same program. - For any string-embedded DSL, search the codebase for concatenation into that string. Every hit is a place the parser was bypassed.
Plausible wrong readings
Stated the way a confident engineer states them.
- "Internal DSLs are just method chaining." Method chaining is one host's way of getting there. The defining property is that the calls build an inspectable program rather than doing the work.
- "External DSLs are more powerful." They have freer syntax. Power comes from semantics, and an internal DSL in a language with macros can be strictly more capable.
- "An internal DSL is a stepping stone and always the wrong final answer." It is frequently the right final answer, especially when the readers are the host language's own programmers.
- "Once I have a parser, the external DSL is done." The parser is the smallest piece of the work — see
[[dsl-tooling-cost]]. - "SQL in a string is an external DSL, so I get the external benefits." You get the costs. The benefits require your own parser and checker, and a string has neither.
Misconceptions
The claim, and what is actually true.
Go deeper
The same idea at increasing depth. Stop wherever it stops being useful.
overview
There are two ways to build a domain language. Write it inside your existing programming language, using function calls and objects that read like domain notation — then the editor, the compiler and the debugger you already have all work on it. Or give it its own file format and syntax, which reads exactly the way the domain wants, and build every tool for it yourself. The second is a much better result and a much larger project.
practical
Ask who reads these programs. If it is the same engineers who work in the host language, go internal — the notation gain is small and the tooling gain is enormous. If it is analysts, operators or domain experts who will not install your toolchain, external is the only option that addresses the problem, and you should budget the tooling in the same plan as the language rather than as a follow-up. Either way, do not let the notation live inside strings in the host language: that combination has every cost of both and no checking at all, and it is where injection bugs come from.
advanced
The most useful reframing is that this is not a syntax decision but a decision about *when the program exists*. Internal DSL programs come into being during host execution, which means the host checked them and can debug them, and means whole-program analysis requires running code first. External DSL programs exist as text before anything runs, which means they can be checked, diffed, planned and reviewed in CI, and means nothing checked them until you wrote the checker. Every practical difference — tooling, audience, analysability, migration cost — follows from that one property. It also explains why the hybrid works so well: an external front end compiling to the internal representation moves the program's existence earlier without giving up the semantics, the tests or the back end that already work, which is the cheapest available route from a stabilised internal notation to a real language.
How much this depends on
Nothing in this domain is true of every compiler. These say how much.
If you were asked this in an interview
- You need a rules language for a fraud team of non-programmers. Internal or external, and why?
- What does an internal DSL get for free, and what does it give up?
- Why is a DSL embedded in strings the worst of both worlds?