By Cheuk Hang Lau, Zeyu Cao, Kevin Wong Cheuk Yin, Yao Lai, Haoran Wu, Nicholas D. Lane, Robert D. Mullins, Ilia Shumailov, Yiren ZhaoSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2609-34683.git
Cite
Version link
https://openversions.com/index/arxiv-2609-34683/commit/3a66c520c80ba1064dc89f91e25b96a71cdce75a
Version identifier (commit hash)
3a66c520c80ba1064dc89f91e25b96a71cdce75a

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2609.34683v1 OpenVersions: 5ec9d905-61dd-4c1d-852d-372af32f9cfd

Posted

Claims

Findings

Headline finding

Agentic LLM workloads occupy a hardware regime that no prior inference benchmark captures, characterized by a dual pressure of compute-bound prefill and KV-cache capacity saturation that separates agentic serving from chat.

Under controlled single-turn comparison at C=80 (matched on model, hardware, backend, and tensor-parallel degree), coding-agent planning calls have a median 2.8x higher TTFT and 1.3x higher TPOT than chat across 40 released trace_replay groups.

At high concurrency (C=200 to C=320), chat workloads continue benefiting from larger batches (median +11% request throughput across 63 groups), whereas agentic workloads are mostly at or past saturation by C=200 and plateau or lose throughput (41 of 66 groups within +/-10% and 15 losing more than 10%).

On a single H100 (LLaMA-3.1-8B, TP=1, vLLM, bf16), coding-agent operational intensity (OI) reaches 2,409 versus 181 for chat at C=80 (a 13x separation), placing agentic workloads above the H100 ridge point (295 FLOP/byte, compute-bound) while chat remains memory-bandwidth-bound.

Per-component roofline decomposition of LLaMA-3.1-8B on H100 (BF16, TP=1) at C=80 shows that prefill projection GEMMs are compute-bound (OI up to 2,956, accounting for 81.8% of modeled layer latency) while attention, elementwise operations, and all decode components remain bandwidth-bound, creating an 8-28x prefill-to-decode OI gap across workloads.

In the analytical capacity footprint model for LLaMA-3.1-8B on a single H100 (80 GB HBM, 16 GB weights), coding-agent sessions exceed the 80 GB HBM wall by C=40 and reach 155 GB at C=80, while terminal sessions reach 126 GB, making KV-cache capacity the phase-level bottleneck.

Using a shared 1024-token synthetic prefix with code-like token structure enables distributional synthetic profiles to reproduce real SWE-Bench trace replay on H100/LLaMA-3.1-8B/vLLM at C=5 with 2.4% aggregate E2EL MAPE and 3.2% turn 10-19 E2EL MAPE.

Methods

Closed-loop concurrency sweep: admitting at most C requests to an OpenAI-compatible endpoint at a time, advancing multi-turn sessions in turn order to preserve context snowballing, and sweeping C to locate saturation C* where |d(req/s)/dC| < epsilon.

Source excerpts

“We evaluate serving with a closed-loop concurrency sweep. At concurrency C, the benchmark harness admits at most C requests to the endpoint at a time. When one request completes, the next request or session turn is dispatched. For multi-turn profiles, each slot advances a session in turn order, preserving the session-context snowballing that makes agentic prompts grow across turns, increasing prefix-cache pressure.” · Show in paper

“Saturation is the concurrency C* at which req/s ceases to rise (|d(req/s)/dC| < epsilon); we sweep C to locate it for every workload, with the TTFT knee and KV-cache fill fraction corroborating C* but not replacing it.” · Show in paper

Workload profile construction: building trace replay profiles from recorded ShareGPT, SWE-Bench, TerminalBench, and OSWorld traces, and distributional synthetic profiles by sampling per-turn ISL, OSL, and turn-count distributions with a prefix-aware configuration (code-like token structure and shared 1024-token synthetic prefix).

Source excerpts

“Trace replay profiles preserve the exact recorded requests from the source traces. Distributional synthetic profiles sample multi-turn requests from per-turn ISL, OSL, and turn-count distributions fitted to those traces, producing representative workloads that can be reissued across models, hardware, backends, and concurrency settings, without the runtime cost of full replay.” · Show in paper

“Adding a shared 1024-token synthetic prefix brings deep-turn E2EL MAPE to 3.2% and aggregate E2EL MAPE to 2.4%, within the observed noise scale. Hence synthetic distributional profiles in this paper use this prefix-aware configuration.” · Show in paper

Kernel-level multi-dimensional roofline profiling: decomposing each decoder layer into constituent operations, computing component operational intensity OIj=FLOPsj/DRAMBytesjOI_j = FLOPs_j / DRAMBytes_j, roofline throughput ceiling Pjroof=min⁡(Ppeak,OIj×BWpeak)P_j^{roof} = \min(P_{peak}, OI_j \times BW_{peak}), and latency lower bound tjroof=FLOPsj/Pjrooft_j^{roof} = FLOPs_j / P_j^{roof}, grounded with Nsight Compute (NCU) kernel profiling records.

Source excerpts

“We decompose each decoder layer into constituent operations and compute component-level FLOPs and DRAM bytes from model dimensions, precision, tensor-parallel degree, active token count, and batch size. For component j, the operational intensity is OIj = FLOPsj/DRAMBytesj, and the roofline throughput ceiling is: Proof_j = min (Ppeak, OIj x BWpeak), where Ppeak is peak compute throughput (FLOP/s) and BWpeak is peak memory bandwidth (byte/s) for the target GPU. We convert the ceiling into a component latency lower bound, troof_j = FLOPsj/Proof_j, and aggregate component costs to obtain a layer-level roofline bound.” · Show in paper

“We ground this analytical decomposition with two NCU-backed artifacts in our dataset. per_layer_kernel provides the representative LLaMA-3.1-8B/H100 prefill decomposition used for the layer-level example, while kernels_labeled provides broader per-kernel NCU timing and shape records across four GPU platforms and 11 model architectures (Appendix Tables 6 and 8).” · Show in paper