Throughput: Requests, Packets and Bytes per Second

Bytes per second is what a link carries and what iperf and NIC counters report. Packets per second is what NICs, drivers, switches and the kernel network stack process, and every packet costs roughly the same amount of that work whether it holds 64 bytes or 1,500. Requests per second is what an application serves, and each request may be one packet or a thousand. A system can be saturated in any one of the three while the other two look idle, and the diagnosis depends entirely on which.

A 10 Gbit/s link carries ~812,000 packets/s of full 1,500-byte frames, or 14.88 million packets/s of minimum-size 64-byte frames. A Linux kernel doing ordinary socket I/O manages roughly 1–2 million packets/s per core before offloads, XDP or kernel bypass (DPDK) are needed; small-packet workloads — DNS, game state, VoIP, a chatty RPC — hit the pps wall at a tiny fraction of the link’s bandwidth. The symptom is a NIC at 10 % bytes-utilisation with a CPU core pinned in ksoftirqd, and the fix is bigger packets (batching), more cores handling interrupts (RSS), or bypassing the kernel.

Throughput and latency are distinct but not independent. For any stable system, Little’s law says: average number of requests in flight = throughput × average latency, L = λW. It holds for a single server, a thread pool, a connection pool, a queue, a whole data centre, with no assumptions about distributions. It is the arithmetic behind every capacity question.

Read it in each direction. A service at 1,000 req/s with 200 ms latency has 200 requests in flight at all times; it needs at least 200 concurrent workers, connections or event-loop slots, or it queues. Fix the concurrency and the law gives the ceiling: 50 database connections at 20 ms per query serve at most 2,500 queries/s — and if latency rises to 40 ms under load, the same 50 connections serve 1,250, which is why saturated systems lose throughput exactly when they need it. And a load test that reports 2,000 req/s at 10 ms latency was measuring 20 in flight; production at 800 req/s and 500 ms latency has 400 in flight and a pool of 100 — the pool is the wall, and the CPU never sees the traffic that is waiting for it.

Above a few Mbit/s the sender is TCP, and TCP’s throughput is set by its congestion window, which loss shrinks and which grows by only one segment per RTT in congestion avoidance (Congestion Control: Protecting the Network). The Mathis relation captures the steady state for classic loss-based congestion control: throughput ≈ (MSS / RTT) × (C / √p), with C ≈ 1.22 and p the packet loss rate. Two facts fall out. Throughput is inversely proportional to RTT — the same loss rate hurts a cross-continent connection 100× more than a same-rack one. And it falls with the square root of loss: 0.01 % loss costs a lot, 1 % is catastrophic.

Numbers: MSS 1,460 bytes, RTT 100 ms. At p = 0.001 (0.1 % loss) the ceiling is ~4.5 Mbit/s; at p = 0.01 (1 %), ~1.4 Mbit/s; at p = 0.0001 (0.01 %), ~14 Mbit/s. On a 10 Gbit/s link. This is why a "slightly lossy" Wi-Fi or a link with a marginal optic makes long-distance transfers crawl while local ones barely notice (Packet Loss: Duplicate ACKs, Fast Retransmit and the RTO), and why BBR — which models bandwidth and RTT instead of reacting to loss — exists. It is also why retransmission and loss counters (ss -ti, nstat -a | grep Retrans) belong next to any throughput measurement.

Measure throughput and latency together, always, because either alone hides the other. iperf3 -c host gives raw TCP bytes/s between two hosts (add -P 8 to see whether one stream is window-bound — see Bandwidth vs Latency); iperf3 -u -b 0 -l 64 hammers small UDP packets to find the pps ceiling. ping and ss -ti (rtt:, retrans:, cwnd:) give latency and loss on a live connection. wrk, hey and k6 give requests/s and a latency distribution; report p50, p99 and the concurrency you drove, or the number is meaningless.

The shape to look for: as offered load rises, throughput rises linearly while latency stays flat, then throughput flattens and latency climbs — that knee is capacity, and Little’s law tells you which resource ran out (in-flight = throughput × latency at the knee, compared with your pool sizes). Measuring past the knee produces a throughput number that looks good and a latency number that is unusable, which is how load tests lie.