Codegentarget

Machine Code Encoding

The last translation: `add rax, rbx` becomes the three bytes 48 01 D8. A REX prefix says the operands are 64-bit, one opcode byte says "add", and a ModR/M byte names both registers.

The question

What bytes does an instruction actually turn into, and why is x86 variable-length when ARM is not?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute

What the program is here

A byte sequence in a code section, plus a relocation table naming the bytes whose values are not yet known. This is the first representation the hardware itself can consume — everything before it was a data structure inside a process that has already exited. It exists to answer the fetcher's only question: where does this instruction end and the next one begin?

What this phase may assume or do

The encoder must produce bytes that the target decoder interprets as exactly the instruction requested — no more, since the mapping from mnemonic to encoding is fixed by the ISA specification and there is no room for judgement. Where there is genuine freedom, it is in *which* encoding: many instructions have several, differing in length and in whether an immediate is one byte or four. Choosing among those is legal whenever the chosen form represents the same operands, and it interacts with branch relaxation, where a short jump encoding may be used only if the target is provably within range after all other choices are final.

Key points

add rax, rbx is three bytes: 48 REX.W for 64-bit operands, 01 the add opcode, D8 the ModR/M byte naming both registers.
ModR/M packs three fields into a byte: mod says whether operands are registers or memory, reg names one register, rm names the other.
x86 instructions are 1–15 bytes and their length depends on their own contents, so decoding is sequential and expensive; a micro-op cache exists largely to avoid doing it twice.
AArch64 instructions are exactly 4 bytes, which makes wide parallel decode trivial and makes large constants and long branches awkward.
The assembler cannot encode addresses it does not know, so it emits relocations; and it iterates to a fixed point over branch encodings because instruction lengths depend on each other.

Three bytes, dissected

targetx86-64 only. The REX prefix, the ModR/M byte and the variable-length structure are specific to the x86 family. AArch64 encodes the same addition as a single fixed 32-bit word — add x0, x0, x1 is 0x8B010000, and the 32-bit form add w0, w0, w1 is 0x0B010000 — with register numbers in fixed bit fields and no prefixes at all. RISC-V is similar in spirit with 32-bit base instructions and an optional 16-bit compressed encoding.

Take add rax, rbx. Our encodeSketch produces 48 01 D8, and each byte answers a different question. This is the real encoding — you can assemble the instruction and check it — and taking it apart is the fastest way to see why x86 decoding is what it is.

The 48 is a REX prefix. On x86-64, the base opcode map was inherited from a 32-bit architecture, so 64-bit operand size and access to the eight registers added in the 64-bit extension both have to be signalled by a prefix byte. 48 is REX with the W bit set: operands are 64-bit. Without it, the same following bytes operate on 32 bits.

The 01 is the opcode. In the x86 map, 01 /r is "add r/m64, r64" — add a register to a register-or-memory destination. Note that the *direction* is part of the opcode choice: 03 /r is the same addition with the operands the other way round, which is why an assembler can encode add rax, rbx two different ways and both are correct.

The D8 is the ModR/M byte, and it carries three fields packed into eight bits: mod (2 bits), reg (3 bits), rm (3 bits). D8 is 11 011 000. mod = 11 means both operands are registers rather than memory. reg = 011 is register 3, rbx. rm = 000 is register 0, rax. So: add rbx into rax, 64 bits wide.

add rax, rbx → 48 01 D8

  0x48        0x01        0xD8
  REX.W       opcode      ModR/M
  |           |           |
  |           |           +-- 11 011 000
  |           |               |  |   |
  |           |               |  |   +-- rm  = 0 -> rax   (destination)
  |           |               |  +------ reg = 3 -> rbx   (source)
  |           |               +--------- mod = 3 -> both operands are registers
  |           +-- 01 /r = ADD r/m64, r64
  +-- 0100WRXB with W=1: operand size is 64 bits

  Change one field and you get a different instruction:
    48 89 D8   ->  mov rax, rbx     (opcode 0x89 = MOV r/m64, r64)
    48 01 D9   ->  add rcx, rbx     (rm = 1 -> rcx)
    48 01 F8   ->  add rax, rdi     (reg = 7 -> rdi)
       01 D8   ->  add eax, ebx     (no REX: 32-bit operands)

Why x86 instructions have no fixed length

An x86-64 instruction is between 1 and 15 bytes. It may carry up to four legacy prefixes, then optionally a REX or VEX or EVEX prefix, then one to three opcode bytes, then optionally a ModR/M byte, then optionally a SIB byte for scaled-index addressing, then a displacement of 1, 2 or 4 bytes, then an immediate of 1, 2, 4 or 8. Which of those are present is determined by the ones before them, so you cannot know where an instruction ends without decoding most of it.

This is not a design; it is thirty-five years of compatible extension. The 8086 encoding was designed for a machine where code size mattered enormously and decoding was cheap relative to everything else. Every extension since — 32-bit, SSE, 64-bit, AVX — had to fit into the gaps left in the opcode map, which is why REX exists, why VEX exists, and why there are three ways to encode some additions.

The cost lands on the decoder. Finding instruction boundaries is inherently sequential: you cannot decode instruction two until you know where it starts, which requires decoding instruction one. Modern x86 cores throw enormous resources at this — parallel length-decoders that speculate on boundaries, and a micro-op cache that stores already-decoded operations precisely so the decoder can be bypassed on hot code. The benefit is genuine code density: x86 code is typically 10–20% smaller than AArch64 for the same program, which is real instruction-cache pressure saved.

Fixed-length encoding makes the opposite trade. Every AArch64 instruction is exactly four bytes, so instruction *n* starts at a computable address and the decoder can decode eight of them in parallel with no speculation whatsoever. The cost is that a 64-bit constant cannot fit in a 32-bit instruction, so loading one takes up to four instructions (mov/movk/movk/movk), and every branch has a limited range that the linker may have to bridge with a veneer.

Variable-length versus fixed-length, and who paystarget

	x86-64	AArch64
Instruction lengthtarget	1–15 bytes	Exactly 4 bytes
Finding the next instructiontarget	Requires decoding this one; inherently sequential	Add 4. Any alignment is a valid start.
Decode widthtarget	Expensive: parallel length-decoders plus a micro-op cache to bypass them	Cheap: decode many per cycle with no speculation
Code densitytarget	Better — common operations have short encodings	Worse — a register-to-register move costs the same 4 bytes as anything else
Large constantstarget	A 64-bit immediate fits in one instruction	Up to four instructions, or a load from a constant pool
Disassembly from an arbitrary offsettarget	Ambiguous; starting one byte late yields a different valid instruction stream	Unambiguous at any 4-byte boundary
Consequence for securitytarget	Unintended instructions exist inside instruction bytes, which is what gadget-hunting exploits	No unintended instruction boundaries, which removes that gadget source

What the assembler still cannot know

implementationModern compilers usually skip the assembly text entirely: Clang and GCC both have integrated assemblers that encode instructions directly from the machine IR, and -S exists to print text for humans rather than as a pipeline stage. That matters when debugging, because the text path and the integrated path occasionally differ in macro handling and relaxation decisions, so a bug that reproduces through -S and as may not reproduce through the integrated path.

Encoding is not quite the end. Two things cannot be resolved when an instruction is encoded, and both leave holes for someone else to fill.

The first is addresses outside this object file. A call printf encodes as a call opcode plus a four-byte displacement that the assembler cannot compute, because it does not know where printf will be. It emits zeroes and records a *relocation*: "at this offset, patch a PC-relative 32-bit displacement to the symbol printf". That is the assembler's handover to [[relocations]] and [[what-a-linker-does]].

The second is branch distances within the same section, which produces a small fixed-point problem called relaxation. A short jump encodes its target in one byte and reaches ±127; a near jump uses four. The assembler wants the short form, but whether the target is within range depends on the sizes of the instructions in between, which depend on whether *they* used short forms. So the assembler iterates: assume short where possible, measure, promote anything out of range, and repeat until stable. On AArch64 the analogous problem is a branch beyond ±128MB, which the *linker* fixes by inserting a veneer — a small trampoline — because the instruction length cannot grow.

How it works

The steps, in the order the compiler takes them.

The encoder looks up the instruction form in the ISA encoding table, keyed by mnemonic and operand kinds.
It emits any required prefixes — for x86-64, a REX byte when a 64-bit operand size or a high-numbered register is used.
It emits the opcode bytes for the chosen form, which already encode the operand direction and sometimes a register number.
It builds the ModR/M byte from the addressing mode and the two register numbers, adding a SIB byte when a scaled index is used and a displacement when the addressing mode needs one.
It appends any immediate, choosing the narrowest field the value fits into.
Any operand whose value is unknown — an external symbol, a cross-section address — is emitted as zeroes with a relocation entry describing how to patch it.
Branch encodings are iterated to a fixed point, since promoting one branch to a longer form can push another out of range.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

A hand-written encoder builds the ModR/M byte with reg and rm swapped, and every instruction it emits operates on the wrong registers — silently, because the result is a perfectly legal instruction.
A JIT emits a short branch and later patches in code that pushes the target out of range, so the branch wraps to a wrong address and the process jumps into the middle of an unrelated function.
An instruction from an ISA extension is encoded for a CPU that lacks it, and the process dies with SIGILL on some machines in the fleet and not others.
A disassembler is pointed at an offset that is not an instruction boundary, produces a plausible-looking but entirely fictitious listing, and an engineer debugs against it for an afternoon.
A generated code buffer is written but not flushed from the data cache to the instruction cache on an architecture that does not keep them coherent, and the CPU executes stale bytes.

When it helps

Writing a JIT or a dynamic binary translator, where instructions are assembled at run time and there is no assembler in the loop.
Reverse engineering and exploit analysis, where instruction boundaries and unintended decodings are the whole subject.
Debugging a corrupted binary or a bad relocation, where the bytes and the disassembly disagree and only the bytes are trustworthy.
Understanding why code size differs so much between architectures for the same program.

When it hurts

Almost all ordinary compiler work. The encoding is fixed by the ISA and there is no judgement to exercise; time spent here is time not spent where the decisions are.
Reasoning about performance. Encoding length affects instruction cache pressure and decode bandwidth, and beyond that a shorter encoding is not a faster instruction.

What it costs

Every one of these is paid by something.

Variable-length encoding buys code density — real instruction-cache and memory-bandwidth savings — and costs a sequential, transistor-hungry decoder plus the ambiguity that makes disassembly and gadget-free code hard.
Fixed-length encoding buys trivial wide decoding and unambiguous boundaries, and costs code size and the ability to embed large constants or long branch displacements in one instruction.
Choosing the shortest encoding for every instruction buys size and costs an iterative relaxation pass in the assembler, plus the risk that a later edit invalidates a range assumption.

What else you could do

What a different compiler or language does instead, and when that is better.

Fixed-length RISC encoding (AArch64, RISC-V base): four bytes each, simple decode, more instructions for the same work.
A compressed extension on top of a fixed-length base: RISC-V's C extension adds 16-bit forms of common instructions, recovering much of the density while keeping boundaries determinable from a small prefix check.
Bytecode instead of machine code: a virtual instruction set designed for size and ease of decoding by software rather than hardware — see [[bytecode]] and [[wasm-model]].
VLIW encoding, where several operations are packed into one wide instruction word and the compiler decides which issue slots they occupy; simple hardware, and completely dependent on the compiler's schedule.

See it for yourself

The flag, dump or tool that shows you this directly.

See mnemonic and bytes together: objdump -d file.o prints both columns; objdump -d --no-show-raw-insn hides the bytes when you only want the listing.
Encode one instruction and look: echo "add rax, rbx" | llvm-mc -triple=x86_64 -show-encoding prints the bytes and the ModR/M breakdown.
The other direction: llvm-mc -triple=x86_64 -disassemble <<< "0x48 0x01 0xd8".
Compare architectures: run the same llvm-mc -show-encoding with -triple=aarch64 and watch every instruction become exactly four bytes.
Ours: encodeSketch in src/compilers/sim/codegen.ts produces the REX / opcode / ModR/M breakdown shown in this lesson for register-to-register forms.
The authority when it matters: the Intel Software Developer's Manual volume 2 for x86-64, the Arm Architecture Reference Manual for AArch64.

Plausible wrong readings

Stated the way a confident engineer states them.

"Each instruction is one byte of opcode plus operands." On x86 an instruction may have four prefixes, three opcode bytes, ModR/M, SIB, a displacement and an immediate. On AArch64 there is no opcode byte at all — the operation is a set of bit fields.
"Shorter encodings run faster." They occupy less instruction cache and less decode bandwidth, which can matter. The execution unit does not care how many bytes the instruction took to express.
"A disassembler shows what the CPU will execute." It shows what the CPU would execute starting from the offset you gave it. On a variable-length ISA a different starting offset yields a different and equally valid instruction stream.
"The assembler resolves all the addresses." It resolves the ones it knows. Anything outside the object file leaves a relocation for the linker, which is the whole reason object files have relocation tables.

Misconceptions

The claim, and what is actually true.

Assembly and machine code are two names for the same thing.

Assembly is text with symbolic names and directives. Machine code is bytes. The assembler is a real translation with real decisions in it — which encoding, which branch form, which relocations.

x86 is variable-length because of bad design.

It is variable-length because the 8086 optimised for code density when memory was the scarce resource, and every extension since had to remain decodable alongside it. The trade was reasonable then and the density benefit is still real; the decoder cost is what changed.

The assembler produces the final bytes.

It produces bytes plus holes. Any address it could not know is a relocation, and the linker fills them in — which is why the same object file can be linked into executables at different addresses.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

An instruction is bytes. add rax, rbx is 48 01 D8: one byte saying the operands are 64-bit, one saying "add", and one naming both registers in packed bit fields. On x86 the number of bytes varies from one to fifteen; on ARM64 every instruction is exactly four.

practical

Reach for the bytes when the disassembly and reality disagree — a corrupted binary, a bad relocation, a JIT patching its own code. objdump -d shows bytes and mnemonics together, and llvm-mc -show-encoding will encode a single instruction so you can check a hand-written encoder field by field. Remember that on x86 a disassembly starting at the wrong offset is confidently wrong rather than obviously wrong.

advanced

The encoding choice is one of the few architectural decisions whose consequences reach every layer. Variable-length gives density, which is instruction-cache footprint, which is real performance on large codebases — and costs a sequential decoder expensive enough that Intel and AMD both cache decoded micro-operations to avoid repeating the work. It also creates unintended instruction boundaries inside instruction bytes, which is the raw material for return-oriented programming and part of why fixed-length ISAs have a structurally smaller gadget surface. Meanwhile fixed-length forces constant pools and linker veneers, and pushes complexity into the linker instead of the decoder. Neither side is the right answer; they are different places to put the same complexity.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

targetREX prefixes, ModR/M and SIB bytes, and 1-to-15-byte instructions are x86-64. AArch64 uses fixed 32-bit words with register numbers in fixed bit fields and no prefixes; RISC-V uses 32-bit base instructions with an optional 16-bit compressed form; the two families share nothing at the byte level.

specThe specific bytes here — 0x48 for REX.W, 0x01 for ADD r/m64 r64, 0x89 for MOV r/m64 r64, and the mod/reg/rm layout of ModR/M — are fixed by the Intel and AMD architecture specifications, not by any compiler. They do not vary between implementations, which is exactly what makes them a specification rather than a convention.

simplifiedOur encodeSketch handles register-to-register operands with the eight original registers only. It emits no SIB byte, no displacement, no immediate, and does not handle r8–r15, which would need the REX.R and REX.B bits set rather than just REX.W. A real encoder is a large table plus careful bit assembly.

If you were asked this in an interview

Take add rax, rbx apart byte by byte and say what each field does.
Why can an x86 disassembler produce a completely different listing if it starts one byte earlier, and why can an AArch64 disassembler not?
What does an assembler do when it encounters a call to a symbol defined in another object file?

Connections

Computer Architectureinstruction-decode instruction-set-architecture isa-vs-microarchitecture instruction-cache endianness

Securitysupply-chain-security