← PracticeIntermediatePipelines

The Leak That Was a Queue

Pull up the evidence one item at a time, commit to a diagnosis, and only then see the schedule that actually ran.

What was reported

The ingest service runs out of memory every six to nine hours and restarts. We have been hunting a leak for a week. Heap dumps are dominated by event objects, but every one of them is legitimately referenced — nothing is held by a listener that forgot to unsubscribe, no closure captures anything odd, and the profiler shows no growth in any cache. We raised the heap from 4GB to 12GB and it now OOMs every eighteen hours instead of every seven.
1BlockingQueue<Event> queue = new LinkedBlockingQueue<>();   // no capacity argument
2 
3// HTTP thread, 12k requests/second at peak
4void ingest(Event e) {
5    queue.put(e);          // never blocks: the queue is unbounded
6    respond(202);          // "accepted" — we told them we have it
7}
8 
9// 4 consumer threads
10void consume() {
11    while (running) {
12        Event e = queue.take();
13        enrichAndWrite(e);   // ~440us each: two lookups and an insert
14    }
15}

Evidence

Nothing here is labelled as relevant. Some of it is not.

What is actually happening?