The question this answers
Which arrows on this diagram have a meter on them, and how much data crosses each one?
The product serves media files to users worldwide, replicates its database to a second region for disaster recovery, and runs a set of services that talk to each other across availability zones. All three requirements are legitimate. All three move bytes across a billed boundary, and nobody costed any of them at design time.
A map of which network paths are metered and at what relative rate, so that a topology can be arranged to keep bytes on the cheap paths without giving up the property that made the expensive path necessary.
Five paths, five different rates
Data transfer is priced by the boundary it crosses, and the boundaries form a rough hierarchy. Traffic within a single availability zone is usually free or nearly so. Traffic between zones in the same region is billed, often in both directions, at a modest rate. Traffic between regions is billed at a higher rate. Traffic out to the internet is billed higher still. And traffic *into* the provider is almost always free, which is the asymmetry that makes the whole thing surprising: you can put a petabyte in for nothing and then discover what it costs to read it back out.
The topology below marks the metered paths on an ordinary global design. The one worth staring at is the cross-zone edge between the API tier and the database, because it is the one no architecture review mentions. A multi-zone deployment is correct for availability, and it means a large fraction of every internal request now crosses a billed boundary. A chatty service decomposition multiplies that: calls that used to be in-process function calls are now cross-zone network hops with a meter, and the total can rival the compute running the services.
The other path worth marking is the NAT. Every outbound byte from a private subnet is processed by it and billed per gigabyte on top of whatever the destination charges — see NAT Gateway. Reading from object storage through a NAT rather than through a private endpoint is the classic instance: the data never needed to leave the provider network at all, and it was billed as though it had made a journey.
The bill shape, and the four moves that change it
The panel splits the transfer bill by path. Three items are marked as surprises and they are the three that appear on no architecture diagram: cross-zone traffic between services, NAT processing, and telemetry shipped to an external vendor. Each of them is generated by a decision that was made for a good reason — availability, private networking, better tooling — and none of them was costed when the decision was made.
Four moves change the shape, in rough order of leverage. Put a CDN in front of anything served repeatedly: origin egress bills every request, CDN egress bills the cache miss, and at volume the CDN rate per gigabyte is lower as well. Use private endpoints for provider services so that object storage and managed-service traffic bypasses the NAT meter and stays on the provider backbone. Compress and batch telemetry before it leaves the network, which routinely cuts observability egress by a large factor for very little work. And co-locate chatty components in the same zone where the availability requirement permits it, accepting that this trades some resilience for a real reduction in cross-zone traffic.
A word on the fifth move people reach for, which is a commitment or a discount negotiation. It lowers the rate and changes nothing about the volume. It is worth doing at scale and it is the last step, not the first — reducing the bytes is almost always larger and it also makes the system faster.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Where the bytes are worth moving
Not all egress is waste, and a lesson that treats it as such produces bad architecture. Serving users is the business. Replicating to a second region is a disaster-recovery requirement someone signed off. Cross-zone traffic is the direct consequence of a multi-zone availability design. The question is never "how do we stop moving data" — it is "is each byte crossing this boundary for a reason we would still choose, and is it crossing on the cheapest path that satisfies that reason".
The matrix is the working version of that question. Read it as: for each path, what makes the volume move, and what is the cheaper arrangement that keeps the property you needed. Note how often the cheaper arrangement is also the faster one. A CDN reduces egress cost and latency together. A private endpoint is cheaper and lower-latency than a NAT path. Compressing telemetry cuts cost and pipeline lag. Egress optimization is one of the rare areas where the cost-motivated change and the performance-motivated change are the same change.
One honest caveat about pricing structure. Providers differ substantially — in whether cross-zone traffic is billed one way or both, in whether egress to their own CDN is free, in the size of the free tier, and in how inter-region rates vary by region pair. Some have moved toward free egress for customers leaving. Treat every rate as something to look up for your provider and region, and treat the *hierarchy* — same zone cheapest, internet most expensive — as the durable part.
| Path | What drives the volume | Cheaper arrangement | What you give up |
|---|---|---|---|
| Origin → internet | Every user request served from the origin | A CDN in front, with cache headers that actually allow caching | Cache invalidation complexity, and staleness you must now reason about |
| Zone → zone | Service-to-service calls and database queries across zones | Co-locate chatty pairs; zone-aware routing that prefers a local replica | Some resilience — a zone-local preference is a weaker failure story |
| Region → region | Continuous replication and cross-region reads | Replicate only what recovery requires; compress; asynchronous rather than synchronous | A larger RPO — see RPO & RTO |
| Private subnet → provider service | Object storage reads, managed-service calls through the NAT | A private endpoint for that service | A per-endpoint charge, usually far below the NAT processing it replaces |
| Anything → external vendor | Logs, metrics and traces shipped out continuously | Regional collector that batches, compresses and samples before sending | Some fidelity, and a component you now operate |
| Provider → provider | Multi-cloud data flows | Keep data and its compute on the same side of the boundary | The portability argument that motivated multi-cloud in the first place |
| Origin → CDN | Cache misses fetching from origin | Longer cache lifetimes, better cache keys, origin shielding | Slower propagation of updates |
Key points
- Moving data costs money, not just storing it, and ingress is usually free while egress is not — an asymmetry that hides the cost until you read it back.
- The rate hierarchy is durable even though the rates are not: same zone cheapest, cross-zone, cross-region, internet most expensive.
- Cross-zone traffic between services is the metered path that appears on no architecture diagram and that a chatty decomposition multiplies.
- A NAT gateway bills per gigabyte processed on top of the destination charge; private endpoints remove those flows rather than widening the path.
- A CDN is usually the single largest egress saving available, and it lowers latency at the same time.
- Egress is not automatically waste. Serving users is the business, and cross-region replication is a recovery requirement. Ask whether it is on the cheapest path that satisfies the reason.
- Reduce bytes before negotiating rates. Volume reduction is larger and it makes the system faster.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Transfer is metered at boundary crossings: zone, region, provider network edge, and in some cases the account boundary.
- • Ingress is generally free; egress is charged per gigabyte at a rate that varies by destination class and by region.
- • Cross-zone traffic is billed per gigabyte and, on some providers, in both directions — so a request and its response can each be charged.
- • NAT gateways add their own per-gigabyte processing charge for traffic that also pays whatever the destination costs.
- • CDN egress is metered separately from origin egress, generally at a lower rate at volume, and cache hits generate no origin fetch at all.
- • Private endpoints keep provider-service traffic on the provider backbone, replacing a per-gigabyte NAT charge with a per-hour endpoint charge.
- • Enable flow logs or transfer reports and find out where the bytes actually go before optimizing anything.
- • Put a CDN in front of anything served more than once, and verify the cache hit ratio afterwards — a CDN with a 20% hit ratio is a component that is not doing its job.
- • Add private endpoints for the provider services you read from most; object storage first, essentially always.
- • Compress and batch telemetry at a regional collector before it leaves the network.
- • Check cross-zone traffic between services after any decomposition; the network cost of splitting a service is rarely in the design document.
- • Re-check pricing structure after provider changes — egress pricing has moved more than most line items in recent years.
- • A media file served directly from object storage, going viral, and billing per download at origin rates for a week.
- • A chatty service mesh generating cross-zone traffic that approaches the cost of the compute running the services.
- • Object storage read through a NAT instead of a private endpoint, paying a processing charge for data that never left the provider network.
- • Cross-region replication of everything rather than what recovery requires, including logs and caches that would be regenerated anyway.
- • A cache with a poor hit ratio, so the CDN adds a hop and a bill while origin fetches continue at nearly the original volume.
- • Debug logging shipped to an external vendor, doubling the observability bill and the egress bill simultaneously.
- • Egress scales linearly with users and with payload size, so a response that grows 20% grows the transfer bill 20% forever.
- • Cross-zone traffic scales with the number of internal calls, which grows super-linearly as a system is decomposed into more services.
- • CDN economics improve with volume: higher cache hit ratios and lower negotiated rates both come with scale.
- • Cross-region replication scales with write volume, not read volume, so a read-heavy system replicates cheaply and a write-heavy one does not.
- • The dimension that runs out first is rarely bandwidth capacity — it is the budget, and it does so quietly.
- • Egress volume is an exfiltration signal. An unexplained spike in outbound bytes to an unfamiliar destination is a security event before it is a cost event.
- • A NAT permits outbound to anywhere by default, which is both the cost path and the exfiltration path — an egress allow-list addresses both — see NAT Gateway.
- • Private endpoints reduce cost and attack surface together by keeping traffic off the public internet entirely.
- • Flow logs are the record of what your workloads talked to; retain them, because they are frequently the only evidence in an exfiltration investigation.
- • Charged per gigabyte at the boundary crossed, with a hierarchy from same-zone (cheapest) to internet (most expensive).
- • Ingress is generally free, which is what makes the first large read-back surprising.
- • NAT processing and cross-zone traffic are the two meters most often omitted from cost models entirely.
- • CDN adoption, private endpoints, telemetry compression and co-location are the four levers, in roughly that order of leverage.
- • Every rate here is relative. Actual pricing depends on provider, region pair, destination class, volume tier and commitment, and it changes.
- • Transfer volume by path — internet, cross-region, cross-zone, NAT-processed — rather than as one aggregate number.
- • CDN cache hit ratio, which directly determines origin egress and is the first thing to check when the bill moves.
- • Top talkers from flow logs: which workload, to which destination, how many bytes.
- • Bytes per request over time, which catches a response that grew without anyone noticing.
- • The signal that lies: total bandwidth utilization. It looks healthy at any volume because capacity is elastic; the meter is what is running, not the pipe.
- • Send fewer bytes: compression, pagination, thinner response payloads and appropriately sized images beat every infrastructure change and improve latency too.
- • Cache at the client with proper cache headers, which is free and eliminates the request entirely rather than making it cheaper.
- • Keep computation next to the data instead of moving the data to the computation — a query that returns 1 KB beats a transfer that moves 1 GB to be filtered elsewhere.
- • For a small system serving modest volumes, do nothing. Egress optimization matters above a threshold, and below it a CDN is added complexity.
- • Providers with materially different egress pricing exist, and for genuinely egress-dominated workloads that is a legitimate input to provider choice.
- • A CDN cuts egress cost and latency; costs cache invalidation complexity, staleness reasoning and another component in the request path.
- • Private endpoints cut NAT charges and attack surface; cost a per-endpoint hourly charge and more configuration per service.
- • Co-locating chatty components cuts cross-zone traffic and weakens the zone-failure story that multi-zone deployment was buying.
- • Compressing telemetry cuts egress and adds CPU at the source plus a collector to operate.
- • Reducing cross-region replication scope cuts continuous transfer and raises the recovery point objective, which is a business decision, not an infrastructure one.
The bytes did not change. The path did.
weight per TB: zone=0 private=1 xzone=2 xregion=5 cdn=6 nat=12 peer=12
What people believe, and what is true
Storage is the cost of data.
Storing a terabyte is usually cheap. Serving it repeatedly to users, replicating it across regions, or reading it through a NAT can each exceed the storage line.
Internal traffic is free.
Traffic within one zone is usually free. Cross-zone and cross-region traffic is billed, and multi-zone deployment plus a chatty decomposition makes that a large number.
A CDN is for performance.
It is usually also the largest available egress saving, because it replaces origin egress with cheaper edge egress and eliminates most origin fetches entirely.
We will negotiate a better rate.
Rate negotiation lowers the multiplier and changes nothing about the volume. Reducing bytes is larger, available immediately, and makes the system faster.