CUDA graphs expose the real decode bottleneck
The benchmark showed kernel-launch overhead, not model math, was what finally limited decode throughput
Started 1.41× slower than vLLM. Added continuous batching -> still behind. Added torch.compile -> somehow got worse. Added CUDA graphs -> 452 vs 460 tok/s. Nearly identical. kernel launch overhead is the real bottleneck at decode time, n