Internet Routing: Autonomous Systems and BGP

Inside one organisation, routing is a solved engineering problem: one administrator, one policy, one interior protocol (OSPF or IS-IS) computing shortest paths over a graph everybody agrees on. The internet is not one organisation. It is a federation of autonomous systems (ASes): networks under a single administrative control, each identified by a number. Your ISP is one; Google is AS15169; Cloudflare is AS13335; a university, a hosting company, a large bank each have one. There are around 75,000 active ASes.

Each AS runs whatever it likes internally. Between ASes, every one of them speaks one protocol — BGP, the Border Gateway Protocol — to say "I can reach these prefixes, and here is the list of ASes a packet would traverse to get there." Your packet’s journey is a sequence of AS-internal shortest-path hops stitched together by BGP decisions at each border.

BGP is a path-vector protocol. An announcement is a prefix plus the ordered list of ASes it has passed through (the AS path). When a router forwards an announcement to a neighbour AS it prepends its own number. Loops are detected trivially: if you see your own AS number in a path, discard it. That is the whole mechanism for avoiding the count-to-infinity problem of pure distance-vector routing.

The important thing is what BGP does *not* do: it does not pick the shortest path. Each AS applies policy first. An operator sets a local preference on routes learned from customers (who pay them) above routes from peers (free) above routes from transit providers (whom they pay). Only among routes of equal preference does AS-path length matter, and after that a chain of tiebreakers. The result is that your packet frequently takes a longer path because it is cheaper for the AS deciding, and that is by design, not a bug.

BGP also does not carry any notion of capacity or latency. A path through three ASes with congested links looks identical to one through three ASes with empty ones. Latency-aware traffic steering is done above BGP — by CDNs measuring from clients, by anycast, by DNS — see CDNs: The Networking View.

ASes connect in two commercial shapes. Transit is paid: a smaller network pays a larger one to carry its traffic to and from the whole internet. Peering is settlement-free: two networks with roughly balanced traffic connect directly and exchange only each other’s (and their customers’) routes, saving both of them transit fees. Much peering happens at Internet Exchange Points (IXPs) — a shared switch fabric in a data centre where hundreds of networks plug in and peer with each other over one port. DE-CIX in Frankfurt and AMS-IX in Amsterdam each carry many terabits per second at peak.

These economics produce hot-potato routing: an AS wants to hand traffic to the next AS as early as possible, because carrying it across its own backbone costs money. So the outbound path from A to B leaves A’s network at the exit nearest the source, while the return path leaves B’s network at the exit nearest *its* source. The forward and return paths therefore go through different cities and different links. traceroute shows you only the forward path; the return is invisible from your end and can be the slow one — traceroute: Discovering the Path Hop by Hop and the asymmetric-latency-cross-region challenge deal with exactly this.

Anycast is the one trick you should know at this level: announce the same prefix from many locations, and BGP’s per-router choice sends each client to a nearby instance. The DNS root servers, 1.1.1.1, 8.8.8.8 and CDN edges all work this way. It is routing, not DNS; see CDNs: The Networking View.

BGP was designed for a network of mutually trusting operators and has no built-in authentication of *who may announce what*. A route leak is an AS re-announcing routes it learned from one provider to another in violation of policy — typically a misconfigured customer announcing its transit provider’s full table to a second provider, which now sees a "customer" route and prefers it. The customer’s modest links then receive a continent’s traffic and collapse. A hijack is an AS announcing a prefix it does not own, or a more-specific of someone else’s prefix; by longest-prefix match (The Routing Table and Longest-Prefix Match) that more-specific wins everywhere it is heard. In 2008 a Pakistani ISP’s attempt to block YouTube domestically by announcing a more-specific of YouTube’s prefix leaked to the world and took the site offline for most of the internet for about two hours.

Withdrawals fail too. In October 2021 a configuration change inside Facebook withdrew the BGP announcements for the prefixes hosting its authoritative DNS servers. Within minutes the names facebook.com, instagram.com and whatsapp.com stopped resolving anywhere, because the resolvers could no longer reach the servers that would have answered — a routing failure that presented as a DNS failure, for roughly six hours. The lesson generalises: from the outside, most routing failures look like something else.

The mitigation being deployed is RPKI: cryptographic records of which AS is authorised to originate which prefix, which routers can validate against before accepting an announcement. Coverage is substantial but not universal, so hijacks still happen. From an application engineer’s point of view, the symptoms are the tell: a service reachable from some networks and not others, latency that jumped for one region only, or a traceroute that shows the wrong country.

You cannot choose the path and you cannot see the whole of it, so design for its properties instead. Latency between two regions is not symmetric and not constant; measure round trips in both directions and over time before promising anything. "Reachable" is per source network, so an outage report from one customer and a healthy dashboard can both be true. Multi-region deployments and anycast front doors exist largely because the middle of the internet is outside your control.

When debugging, traceroute/mtr from the affected client and from the server *toward the client* show both halves of the path; a looking glass (public BGP route servers many networks expose) shows what the rest of the world is being told about your prefix. Knowing that a problem is "in the middle" ends the investigation of your own servers quickly, which is most of the value.