Toolingspec

The Language Server Protocol

JSON-RPC over a pipe, and one decision that mattered more than any of its message types: standardizing the interface turned M editors times N languages into M plus N. The gotcha worth knowing is that its positions are UTF-16 code units.

The question

Why did every editor suddenly get good support for every language at roughly the same time?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute
What the program is here

A message stream, not a program representation. The protocol's model of the program is deliberately thin: documents identified by URI and version, positions as a zero-based line and character pair, ranges as two positions, and everything semantic — symbols, completions, diagnostics, edits — expressed as plain JSON structures over those coordinates. The whole design bet is that a *coordinate system plus a vocabulary of requests* is enough shared representation, and that everything else can stay private to each server.

What this phase may assume or do

A client may assume nothing about a server beyond what the server declared in its initialize response. Capabilities are negotiated, not assumed: a client that sends textDocument/rename to a server that did not advertise renameProvider is out of contract, and a server must not send a notification the client did not say it supports. The second precondition is on positions: both sides must agree on how a character offset is counted, which defaults to UTF-16 code units regardless of the encoding of the file — so a server whose internal representation is byte offsets must convert on every boundary crossing, or every span it reports is wrong the first time a non-ASCII character appears earlier on the line.

Key points

  • LSP standardizes the editor-to-language-tool boundary, turning M editors × N languages into M + N implementations.
  • The transport is JSON-RPC 2.0 over stdio with Content-Length framing; a server is an ordinary subprocess.
  • initialize negotiates capabilities in both directions, so feature availability is the intersection of what client and server implement.
  • Diagnostics are pushed by the server as unsolicited notifications, because only the server knows when analysis finished.
  • Expensive completion detail is deferred to completionItem/resolve to stay inside the interaction latency budget.
  • Position.character counts UTF-16 code units by default, so a byte-offset frontend must convert at every protocol boundary.
  • Non-BMP characters are two UTF-16 code units, so a test suite without emoji will not catch the conversion bug.
  • The protocol is a lowest common denominator and it won anyway, because interface standards are worth more by adoption than by quality.

M×N becomes M+N

Before the protocol, editor support for a language was a bespoke integration. Making Rust work in VS Code, in Vim, in Emacs and in Sublime meant four plugins, each embedding or shelling out to something Rust-specific, each with its own idea of completion and navigation, and each maintained by someone different. Adding a language meant repeating that for every editor; adding an editor meant repeating it for every language. Four editors and five languages is twenty integrations, and the quality of any given cell in that grid depended on whether one enthusiast happened to care about that pair.

The Language Server Protocol's contribution is not any of its messages. It is the decision to put a documented interface at that boundary, so a language implements one server and every conforming editor gets it, and an editor implements one client and every conforming language works in it. Twenty integrations becomes nine. At real scale — dozens of editors, hundreds of languages — the difference is between "some pairs work" and "essentially all of them do".

It is the same argument as any standard interface, and worth recognising as such: the protocol is a [[dsl]]-shaped decision about where to draw a boundary, and its value comes almost entirely from being *the* boundary rather than from being a particularly good one. Several of its design choices are widely regarded as mistakes, and it won anyway, because the alternative to a flawed common interface is no common interface.

The integration count, before and aftertypical
SetupIntegrations to buildAdding one editor costsAdding one language costs
Bespoke plugins, 4 editors × 5 languages205 new plugins4 new plugins
LSP, 4 editors + 5 languages91 client1 server
LSP, 20 editors + 100 languages120 instead of 20001 client1 server

What is actually on the wire

The transport is deliberately boring, and being boring is the point: JSON-RPC 2.0 messages over a stream — usually the server's stdin and stdout — each framed by an HTTP-style Content-Length header, a blank line, and the JSON body. No sockets required, no service discovery, no serialisation format to argue about. A server is a subprocess that reads and writes.

The session has a fixed shape. The client sends initialize with its capabilities and the workspace roots; the server replies with *its* capabilities, which is how the client learns whether this server does rename, or code actions, or semantic tokens, and in what form. The client sends initialized, and from then on the traffic is documents and queries.

Two directions matter. Client to server: textDocument/didOpen with the full text, then didChange with either the whole document or incremental ranges depending on what the server asked for in its sync options, then requests — completion, hover, definition, references, rename, codeAction, formatting. Server to client: textDocument/publishDiagnostics, which is a *notification* rather than a response, sent whenever the server has something to say. That asymmetry is deliberate and important: diagnostics are pushed, because the server is the only party that knows when analysis finished.

The other design element worth knowing is laziness. completion returns a list where each item may be missing its documentation and its detailed type; the client asks for those with completionItem/resolve only for the item the user highlighted. Computing full detail for two thousand completion candidates would blow the latency budget, so the protocol splits the work. The same pattern appears in code lens (codeLens/resolve) and code actions.

A hover request and its reply, framing included
1Content-Length: 152\r\n\r\n
2{"jsonrpc":"2.0","id":7,"method":"textDocument/hover",
3 "params":{"textDocument":{"uri":"file:///p/src/main.rs"},
4           "position":{"line":41,"character":12}}}
5 
6Content-Length: 168\r\n\r\n
7{"jsonrpc":"2.0","id":7,
8 "result":{"contents":{"kind":"markdown","value":"```rust\nfn parse(s: &str) -> Result<Ast, Error>\n```"},
9           "range":{"start":{"line":41,"character":8},
10                    "end":{"line":41,"character":13}}}}

Everything interesting is in position. Line 41 is zero-based. Character 12 is *not* a byte offset and *not* a count of user-perceived characters — by default it is a count of UTF-16 code units from the start of the line. A server holding the file as UTF-8 bytes must convert in both directions, and getting that wrong is the most common interoperability defect in the ecosystem.

The UTF-16 gotcha

specThis is a specification requirement rather than an implementation quirk: LSP defines Position.character as an offset in UTF-16 code units, and 3.17's positionEncoding client capability makes it negotiable rather than removing the default. The consequence is that identical source bytes produce different character values under different negotiated encodings, so a position is only meaningful together with the encoding it was computed under. Neighbouring protocols differ: the Debug Adapter Protocol uses one-based lines and columns with its own encoding rules, so positions cannot be passed between the two without conversion.

This deserves its own section because it catches essentially everyone, and because the symptom looks like an off-by-one bug rather than an encoding bug.

LSP inherited its position model from JavaScript, where a string is a sequence of UTF-16 code units and s.length counts them. So Position.character is defined as an offset in UTF-16 code units from the line start, *regardless* of the file's actual encoding. For ASCII text this is identical to a byte offset and to a character count, which is why the bug hides. Introduce one non-ASCII character earlier on the line and the three diverge: é is one code unit and two UTF-8 bytes; → is one code unit and three bytes; an emoji outside the Basic Multilingual Plane such as 😀 is two UTF-16 code units — a surrogate pair — and four UTF-8 bytes.

A compiler frontend almost always works in byte offsets, because that is what a lexer produces and what a span is. So every position crossing the protocol boundary needs conversion, in both directions, per line. The naive implementations get it wrong in one of two ways: treating character as a byte offset, which makes every diagnostic after a non-ASCII character land a few columns early; or treating it as a Unicode scalar count, which is correct until somebody uses an emoji and then is off by one per emoji.

LSP 3.17 added a negotiated positionEncoding capability, so a client and server may agree on utf-8, utf-16 or utf-32 at initialize time. It is a genuine improvement and it does not remove the problem, because the default remains utf-16 and a server must support that to work with clients that did not implement the extension. The practical rule is: keep byte offsets internally, convert at the protocol edge, and put a non-ASCII character in your test fixtures. A test suite of pure-ASCII files will pass forever with this bug in it.

What it gets wrong, and why it won anyway

A standard interface is a compromise by construction, and it is worth knowing this one's.

It is a lowest common denominator. Features that a specific editor and a specific language could do beautifully together get flattened into a generic request. IntelliJ's Java support and Visual Studio's C# support both predate LSP and both remain deeper than a protocol-mediated equivalent, because they are free to invent interactions the protocol has no vocabulary for. This is the standard cost of standardising, and it is real.

The protocol grows. Semantic tokens, inlay hints, call hierarchy, type hierarchy, inline values, notebook documents, pull-model diagnostics — each version adds capabilities, so "supports LSP" describes a moving and partial target. In practice a client and server negotiate down to the intersection of what both implement, which means feature availability is a matrix rather than a yes or no.

Positions are a weak shared representation. Line and character coordinates are fragile against concurrent edits: a server computes a rename edit against document version 12, the user types while it works, and the client must either apply the edit against the version it was computed for or reject it. The protocol carries document versions for exactly this reason, and clients that ignore them corrupt files.

None of that stopped it, and the reason is worth internalising as a design lesson rather than a piece of history: the value of an interface standard is superlinear in adoption and only weakly related to its quality. A better protocol with a tenth of the adoption would have produced a tenth of the ecosystem. The same argument explains the Debug Adapter Protocol, which did the identical thing for debuggers and is why [[debug-information]] consumers became editor-agnostic at the same time.

  • Capabilities are negotiated at initialize; assume nothing not advertised, on either side.
  • Diagnostics are pushed by the server as notifications — LSP 3.17 added a pull model as an alternative, so both exist.
  • Expensive fields are deferred: completionItem/resolve fills in documentation only for the item the user looked at.
  • Document versions accompany every change and every edit; ignoring them is how a rename corrupts a file being typed in.
  • Positions are UTF-16 code units unless positionEncoding was negotiated otherwise. Test with non-ASCII text.
  • The Debug Adapter Protocol is the same idea for debuggers, with different conventions — one-based lines among them.

How it works

The steps, in the order the compiler takes them.

  • The editor launches the language server as a subprocess and speaks JSON-RPC over its stdin and stdout, framing each message with a Content-Length header.
  • The client sends initialize with its own capabilities and the workspace folders; the server responds with its capabilities, including which requests it handles and which document-sync mode it wants.
  • The client sends textDocument/didOpen with the complete buffer, making the server's copy authoritative and independent of what is on disk.
  • Each edit produces a didChange notification — full text or incremental ranges — carrying a monotonically increasing document version.
  • The server analyses and pushes publishDiagnostics when it has results, unprompted, tagged with the version it analysed.
  • User actions produce requests carrying a document URI and a position; the server converts the position into its internal offset representation, answers, and converts any spans back.
  • Results that would be expensive to compute for every candidate are returned incomplete, with a follow-up resolve request for the one the user selected.
  • Edits are returned as a WorkspaceEdit — a set of text edits per document, optionally version-stamped — which the client applies atomically or rejects if the document has moved on.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • Every diagnostic on a line containing an accented character is underlined a column or two to the left, because the server treated character as a byte offset.
  • Highlighting drifts by one column per emoji on the line, because the server counted Unicode scalars rather than UTF-16 code units.
  • A rename computed against an older document version is applied after the user kept typing, and the edit lands in the wrong place, corrupting the file.
  • A client sends a request the server never advertised, and the server either errors or silently does nothing, presenting as "this feature does not work in this editor".
  • Completion feels slow because the server computes full documentation for every candidate instead of deferring it to resolve.
  • A feature works in one editor and not another for the same language, because the two clients implement different subsets of the protocol version.
  • The server writes a stray line to stdout — a debug print — and corrupts the framed message stream, killing the session with a parse error that names nothing useful.

When it helps

  • Any new language that wants tooling: one server buys presence in every editor, which is the difference between a language people can try and one they cannot.
  • Any editor that wants breadth: one client implementation buys the entire existing server ecosystem.
  • Organisations with a heterogeneous editor population, where a single server means everyone gets the same diagnostics and the same navigation regardless of what they use.
  • Internal DSLs and config formats, where a small server providing diagnostics and completion is a few hundred lines and transforms usability — see [[dsl-tooling-cost]].

When it hurts

  • When the language and editor pairing could support a genuinely richer interaction than the protocol can express, and the generic vocabulary flattens it.
  • For features needing very tight interaction loops — sub-frame feedback during typing — where a subprocess and a JSON round trip are the wrong architecture.
  • When protocol-version drift means a feature exists in the server and not in the client, producing support matrices that users experience as random.
  • For anything requiring shared state richer than positions and text edits, where the coordinate model forces awkward encodings.

What it costs

Every one of these is paid by something.

  • A standard interface buys M+N instead of M×N and pays a lowest-common-denominator feature set that no pairing can exceed without extensions.
  • JSON over stdio buys universal implementability with no dependencies and pays serialisation cost on every message plus the fragility that any stray stdout write corrupts the stream.
  • Capability negotiation buys forward and backward compatibility and pays a combinatorial support matrix that users see as inconsistent behaviour between editors.
  • UTF-16 positions bought alignment with the JavaScript ecosystem the protocol came from and pay a conversion at every boundary for every server not written in JavaScript — forever, since the default cannot change without breaking clients.
  • Deferring expensive fields to resolve buys completion latency and pays an extra round trip plus two code paths for building the same item.
  • Running the server as a separate process buys crash isolation and language independence and pays process memory, start-up latency and an IPC hop on every query.

What else you could do

What a different compiler or language does instead, and when that is better.

  • An editor-specific plugin API, which is what IntelliJ and Visual Studio use for their first-party languages. Deeper integration, better latency, and one implementation per editor — the cost this protocol exists to avoid.
  • A library rather than a protocol: link the analysis into the editor directly. Fastest possible, and it requires every editor and every language to share a runtime, which is why it only happens within a single vendor's stack.
  • Purely syntactic tooling via tree-sitter grammars, which gives highlighting, folding and structural navigation for many languages with no server at all — and no semantics.
  • Batch tools invoked on save with parsed output, which is what many editors still use for linters. Simple, robust, and limited to diagnostics.
  • A richer protocol designed for a single ecosystem — the Build Server Protocol and the Debug Adapter Protocol are siblings doing this for builds and debugging respectively, and they show the pattern generalising rather than an alternative to it.

See it for yourself

The flag, dump or tool that shows you this directly.

  • Read the traffic. In VS Code set "<language>.trace.server": "verbose" and open the corresponding Output channel; in Neovim use :LspLog after vim.lsp.set_log_level("debug"). Almost every "the IDE is wrong" question is answered by reading twenty lines of this.
  • Drive a server by hand: run clangd or rust-analyzer in a terminal and paste framed JSON-RPC messages into stdin. Doing this once for initialize plus textDocument/didOpen plus hover makes the whole protocol concrete in about ten minutes.
  • Server-side logs: clangd --log=verbose, RA_LOG=lsp_server=debug rust-analyzer, gopls -rpc.trace, and "typescript.tsserver.log": "verbose" for tsserver.
  • The specification itself is readable and is the authority on every message shape: microsoft.github.io/language-server-protocol, with the version history showing exactly when each capability was added.
  • For the position-encoding bug specifically: create a fixture file whose first line is let x = "héllo 😀"; // comment and check that a diagnostic on the comment lands where you expect. That one line catches both common conversion errors.
  • Existing server implementations as references: vscode-languageserver-node for TypeScript, tower-lsp for Rust, pygls for Python — each handles framing, dispatch and position conversion so you can read what the edge cases actually are.

Plausible wrong readings

Stated the way a confident engineer states them.

  • "LSP is a language server." It is the protocol. The server is the program that implements it, and everything difficult — incrementality, error tolerance, the analysis itself — lives there, not in the protocol.
  • "If a server speaks LSP, the editor experience will be good." The protocol fixes the messages and says nothing about latency, correctness or which capabilities are implemented. A conforming server can recompile the world on every keystroke.
  • "Positions are character offsets." They are UTF-16 code-unit offsets by default. Those coincide only for text in the Basic Multilingual Plane, and diverge from bytes as soon as anything is non-ASCII.
  • "Diagnostics are returned when the editor asks." They are pushed by the server as notifications in the original design, because only the server knows when the analysis finished. A pull model was added later and both are in use.
  • "The protocol solved editor tooling." It solved the *integration* problem. Building a server that is fast and correct is the hard part and it is entirely unchanged by the protocol existing.

Misconceptions

The claim, and what is actually true.

The protocol determines how good the tooling is.
It determines whether the tooling is reachable from your editor. Quality is decided by the server's incrementality, error tolerance and correctness, none of which the protocol constrains.
UTF-16 positions are a legacy detail that no longer matters.
They remain the default in every version, so a server must implement the conversion to interoperate. It is the most common source of off-by-a-few span bugs in the ecosystem.
One protocol means all editors behave the same.
Behaviour is the intersection of what a given client and server implement at their respective protocol versions, which differs per pairing and changes over time.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

Before this protocol, making a language work well in an editor meant writing a plugin for that specific editor, and doing it again for the next one. The Language Server Protocol puts a documented interface in between: the language team writes one server, the editor team writes one client, and every combination works. That is the whole idea, and it is why support for new languages appeared everywhere at once rather than one editor at a time. Underneath it is unglamorous — JSON messages over the program's standard input and output.

practical

If you are writing a server, three things will save you days. Use an existing framework (tower-lsp, pygls, vscode-languageserver-node) rather than implementing framing and dispatch yourself. Keep byte offsets internally and convert only at the protocol edge, with a line-index structure so the conversion is not a linear scan per request. And put a non-ASCII character and an emoji in your very first test fixture — an all-ASCII test suite will pass with the position bug in it, and you will find out from a user in a language you do not read. If you are debugging someone else's server, turn on the client trace and read the initialize response first: most missing features are simply never advertised.

advanced

The interesting property is what the protocol chose *not* to standardise, and how load-bearing that omission is. It fixes the message vocabulary and the coordinate system and says nothing about incrementality, caching, project model, build configuration or how a server should behave under concurrent edits. That omission is what let servers with radically different architectures — a query-driven one, a preamble-caching one, a versioned-reuse one — all conform, and it is also why conformance predicts almost nothing about quality. The comparison worth drawing is with the Debug Adapter Protocol, which made the same bet at the same boundary for debuggers, and with the Build Server Protocol for build systems: three standards, one pattern, each converting a quadratic integration problem into a linear one by naming the interface. The pattern generalises to any ecosystem with two independently-evolving sides and no incumbent monopoly, and the recurring lesson is that being first and adequate beats being late and correct.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

specPositions are specified as UTF-16 code-unit offsets, with positionEncoding negotiation added in 3.17 as an option rather than a replacement. Line numbers and character offsets are both zero-based. The Debug Adapter Protocol, which is the sibling standard for debuggers, uses one-based lines and columns by default and negotiates that separately, so positions are not interchangeable between the two protocols even within one editor session.
implementationCapability coverage differs sharply between clients and between servers, and "supports LSP" is not a useful predicate. VS Code implements essentially all of it; other clients implement subsets that vary by version and by plugin. The same is true of servers. When a feature is missing, the first check is which side advertised what in the initialize exchange, which is visible in the trace log.
typicalRunning the server as a subprocess over stdio is the standard deployment and what most clients assume, but the protocol is transport-agnostic: TCP sockets and web sockets are both used, notably for browser-based editors and remote development. The framing and the message semantics are unchanged; only the pipe differs, which is why a server written against stdio usually needs no changes to be hosted differently.

If you were asked this in an interview

  • Explain the M×N argument for a protocol like LSP in two sentences.
  • A server reports diagnostics two columns to the left on lines containing accented characters. What is wrong?
  • Why are diagnostics pushed by the server rather than requested by the client?

Connections

DSAhash-tablebinary-tree
Performancelatency-fundamentals
API Designversioningbackward-compatibilitydeprecation
Domains that do not exist yet
  • Programming Languages & Runtime Internals — Text encoding in memory: UTF-8 byte strings versus UTF-16 code-unit strings
    The protocol's position model is a direct consequence of JavaScript's in-memory string representation, and the conversion cost lands on every server whose language stores text differently. How runtimes represent strings and what indexing them costs is owned there; why a compiler must convert at the protocol boundary is ours.
  • DevOps / Production Engineering — Shipping and versioning a developer tool that must interoperate with clients you do not control
    A language server is a distributed system with a version-skew problem: clients update on their own schedule and negotiate down to an intersection. Managing that compatibility surface across releases is a delivery-engineering concern owned there; what the negotiation is about is ours.