intermediate

The Kernel Is Fast and the Function Is Not

Read the counters before the options. Nothing here is labelled with the answer.

The report

We moved our image filter to the GPU. The kernel benchmarks at 0.8 ms against 11 ms on the CPU — a 14× win. But the endpoint that calls it got slower, not faster. The GPU is clearly doing its job, so we assume the overhead is somewhere in our web framework.

The per-request path
function filter_image(img):        // img ≈ 12 MB
    d_in  = device_alloc(size(img))
    copy_to_device(d_in, img)          // host -> device
    d_out = device_alloc(size(img))
    launch(filter_kernel, d_in, d_out)
    synchronize()
    result = copy_from_device(d_out)   // device -> host
    device_free(d_in); device_free(d_out)
    return result
CountersSIMULATED
kernel execution time0.8 msThe kernel itself completes in well under a millisecond.
host-to-device copy2.1 msMoving the input to device memory takes over twice the kernel time.
device-to-host copy2.3 msMoving the result back takes a similar amount again.
device allocation + free3.9 ms per callAllocating and releasing device buffers is a per-call cost of several milliseconds.
launch + synchronize overhead0.6 msDispatching the kernel and waiting for completion has a fixed cost.
end-to-end GPU path9.7 ms (CPU path: 11 ms)The complete GPU path is marginally faster than the CPU path it replaced.
What is the hardware doing?
The Transfer You Forgot to CountCPU or GPU: Two Bets About What Work Looks LikeWhat GPU-Friendly Work Has in CommonLanes, Divergence and CoalescingPCIe: Lanes, Generations and the Transfer BudgetThe Specialization Trade-off