By Wen Huang, Jack LiuSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2611-13412.git
Cite
Version link
https://openversions.com/index/arxiv-2611-13412/commit/290d282190690832f3b47c8c6b699dc31b844bb0
Version identifier (commit hash)
290d282190690832f3b47c8c6b699dc31b844bb0

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

Prefix-cached LLM agent serving is metastable: a transient trigger such as a few minutes of extra arrivals at a base load the cache-warm server otherwise sustains can push the server into a persistent degraded regime with low hit ratio, halved throughput, and a growing backlog that does not recover after the trigger ends.

On vLLM with Qwen3-8B at λ=0.035 sessions/s, a 5-minute +1.5λ burst causes persistent degradation in 9 of 12 paired runs (0 in the reverse direction, p=0.00391), whereas the persistent regime is absent without prefix caching or with a session cap.

Persistent degradation also appears with Qwen3-32B on its full KV pool replaying SWE-bench agent trace shapes and on SGLang 0.5.22, though SGLang tolerates larger triggers than vLLM.

In the degraded regime, nearly every turn misses and the server runs at its no-cache capacity; a first-order workload-balance condition λ<1/S\lambda < 1/S, with per-miss cost c(L)c(L) calibrated independently on cache-disabled runs, brackets when the regime can drain.

Once collapsed, suspending admitted sessions at turn boundaries against a limit derived from the KV pool (the budgeter or a tuned session cap) restores throughput and a 0.93–0.95 hit ratio within 6 minutes in every collapsed run, whereas throttling new admissions alone recovered only 1 of 7 matched collapsed runs.

A budgeter that bounds the resident KV demand D=∑s∈A(Ls+os)D = \sum_{s \in \mathcal{A}}(L_s + o_s) of active sessions to a hysteresis band of the KV pool (0.75C0.75C to 0.95C0.95C) without per-workload tuning prevented collapse in every tested run and adapted to session-mix shifts where fixed session caps either collapsed or over-throttled.

A CPU KV offload tier delays the collapse but does not prevent it.

Methods

Paired-run experimental design on vLLM 0.20 (ROCm) using single AMD MI300X GPUs serving Qwen3-8B with a 400K-token KV pool, where twin runs share an identical base Poisson arrival stream of synthetic 25-turn closed-loop sessions and differ only in a 5-minute +1.5λ burst injected at minute 15.

Source excerpts

“We run vLLM 0.20 (ROCm build 0.20.2rc1.dev253) with automatic prefix caching on single AMD MI300X GPUs, one engine per GPU, BF16 weights and KV cache, default scheduler (chunked prefill, FCFS), 40K maximum context, greedy decoding with a fixed output length per turn... Both runs of a pair share one base arrival stream: every base session arrives at the same time with the same identity and shape. The burst run additionally receives an independent stream of extra sessions (rate 1.5λ for 5 minutes, i.e. 2.5× total arrivals) starting at minute 15.” · Show in paper

Whole-session trace-shape replay using 200 trajectories sampled from public mini-SWE-agent leaderboard runs on SWE-bench Verified, preserving initial context, per-turn output, per-turn context growth, and inter-turn gaps (truncated at 38K tokens) against Qwen3-32B on its automatically sized KV pool (≈365K tokens).

Source excerpts

“We replay whole-session shapes—initial context, per-turn output, per-turn context growth and inter-turn gaps—of 200 trajectories sampled from public mini-SWE-agent leaderboard runs on SWE-bench Verified by six 2026 models” · Show in paper

“against Qwen3-32B with its KV pool sized automatically at 80% GPU-memory utilization (≈365K tokens, no explicit cap).” · Show in paper

Independent calibration of cold-turn cost c(L) by regressing window length on the number of completed turns and total prompt tokens across 48 complete two-minute windows with at least four running requests on prefix-cache-disabled runs.

Source excerpts

“We measure c(L) independently, on runs with prefix caching disabled: from 48 complete two-minute windows in which at least four requests were running, we regress window length on the number of completed turns and their total prompt tokens, giving c(L) = −0.06 s + 0.146 ms × L (tokens; the intercept is a fit artifact), hence S ≈ 78 s and 1/S ≈ 0.013 sessions/s for the synthetic workload” · Show in paper

Gateway-level session admission and turn-boundary parking controller (comparing budgeter, session cap + park, adaptive cap + park, latency AIMD + park, and admission-only cap) implemented without engine changes, parking sessions in a FIFO queue when the limit is exceeded and re-admitting parked sessions before new ones.

Source excerpts

“We implement all controllers at the gateway (no engine changes) with the same actuator, so that they differ only in how they set the limit. Before each turn of a session, the gateway decides whether the session may proceed. A session that is not admitted, or that would push the active population over the limit, is parked at this turn boundary and waits in a FIFO queue. Parked sessions are re-admitted before any new session, one at a time as the limit allows” · Show in paper