Benchmarking Tiered KV Cache Offloading for Long-Context AI

Long-context artificial intelligence workloads place severe demands on infrastructure, as agentic applications, multi-turn conversational assistants, and retrieval-augmented generation pipelines repeatedly process extensive context windows. When prompt prefixes are resent across turns, efficient reuse depends on the Key-Value cache. However, GPU high-bandwidth memory is severely limited, running out long before long-context windows are exhausted, which forces systems to discard cached states and recompute them from scratch. To mitigate this memory bottleneck, platform engineers can implement kv cache offloading, extending the cache hierarchy from GPU high-bandwidth memory into CPU DRAM, local NVMe, and distributed file systems.

Evaluated on Red Hat OpenShift AI 3.5 using vLLM 0.24.0 and llm-d 0.8.0 on an OpenShift 4.22.2 cluster with 8xH100 nodes, tiered architectures fundamentally alter the economics of long-context inference. The performance gains depend directly on whether a model’s working set fits within the CPU offload tier or overflows into secondary storage. Understanding the architectural trade-offs, storage latency profiles, and restore-versus-recompute crossover thresholds is essential for designing resilient, high-throughput artificial intelligence platforms.

Architectural Imperative for Tiered KV Cache Offloading

Inference execution is split into two distinct phases with radically different resource profiles. Prefill processes the entire input prompt in a single forward pass where every token attends to every preceding token. This phase is compute-bound, scales quadratically with context length, and directly dictates the time to first token. Decode subsequently generates output tokens autoregressively. Each new token attends to all prior keys and values but computes its own query, making decode memory-bandwidth-bound and comparatively inexpensive per token.

The core constraint is physical capacity. The Key-Value cache footprint scales linearly with context length. As high-concurrency requests saturate GPU high-bandwidth memory, the system must either evict blocks and force costly recomputation later or offload evicted blocks to cheaper memory and storage tiers.

Anatomy of the KV Cache and Working-Set Dynamics

To understand why offloading is mandatory, one must calculate the exact byte footprint generated by modern transformer architectures. For a given tensor-parallel rank, the memory required per token is defined by the following formula:

bytes_per_token = 2 * L_attn * (H_attn / TP) * d_head * b_kv

In this calculation, $L_{attn}$ represents the number of attention layers, $H_{attn}$ is the number of Key-Value heads, $TP$ is the tensor-parallel degree, $d_{head}$ is the head dimension, and $b_{kv}$ is the precision in bytes (accounting for the factor of 2 for both keys and values). For instance, Llama 3.1 70B in FP8 requires approximately 5 MB per 16-token block server-wide, translating to roughly 0.3 MB per token. A single 128K-token context consumes about 40 GB of Key-Value cache before factoring in concurrent requests or model weights.

As concurrent requests execute, they access a subset of cached blocks known as the active working set. When this working set exceeds the capacity of the immediate tier, retention and eviction policies dictate which blocks remain in memory. If an evicted block is requested again and cannot be found in any offload tier, the graphics processing unit must recompute the entire prefix.

Characterising Storage Latency Profiles Using FIO

Evaluating storage tiers using standard sequential bandwidth metrics is insufficient. Sequential 1 MB reads measure bandwidth ceilings rather than the random input-output access patterns characteristic of vLLM Key-Value cache connectors. The engine manages fixed 16-token blocks, performing direct synchronous writes alongside concurrent reads and writes. The definitive metric for storage viability is per-operation latency, particularly p99 tail latency, evaluated under matching block sizes and concurrency levels.

Storage performance can be characterised using `fio` with parameters tuned to match connector patterns. According to the Red Hat Developer analysis (available at Red Hat Developer), the following command configuration uses direct input-output, 16 jobs, and a mixed 75/25 read/write ratio:

fio --rw=randrw --rwmixread=75 --bs=<model_block_size> --iodepth=1 --numjobs=16 --direct=1 --ioengine=libaio --runtime=60

Empirical testing across storage classes reveals stark differences in latency profiles:

Local PCIe Gen 4 NVMe provides the most stable and lowest latency, making it ideal for single-node cache locality. Raw VPC block storage maintains predictable p99 latencies close to p50, whereas CephFS introduces protocol overhead and higher tail latencies. RGW/S3 object storage exhibits extreme p99 tails exceeding 1.1 seconds for 5 MB blocks, introducing severe margin and service level objective risks.

Calculating the Restore-Versus-Recompute Crossover Point

Offloading is only economically viable if restoring a Key-Value block from storage incurs lower latency and compute cost than recomputing the prefix on the graphics processing unit. The crossover context length ($L_{crossover}$), the point where storage restoration and prefill recomputation cost match, is calculated as:

L_crossover = T_FIO,p99 / t_prefill

Where $t_{prefill}$ equals time to first token divided by the context length, and $T_{FIO,p99}$ represents the p99 block retrieval latency.

* Local NVMe: With low p99 latencies (e.g. 14.1 ms for Llama 3.1 70B blocks), $L_{crossover}$ occurs at very short context lengths (under 100 tokens), making offloading universally favourable.
* CephFS: Higher p99 latencies (e.g. 451 ms) push crossover thresholds to thousands of tokens (approx. 2,255 tokens for Nemotron-253B), making shared file system offload best suited for long-context retrieval-augmented generation and agentic workloads.

Inference Benchmarking Topology and Workload Setup

The evaluation environment comprised Red Hat OpenShift AI 3.5, integrating vLLM 0.24.0 and llm-d 0.8.0 on an OpenShift 4.22.2 cluster configured with 8xH100 nodes. Three models were evaluated: Qwen3.6-35B-A3B (MoE) on 2 GPUs, Nemotron-Ultra-253B FP8 on 8 GPUs, and Gemma 4 31B on 2 GPUs.

Each deployment allocated 300 GB of shared system memory for the CPU offload tier, with connector parameters (`cpu_bytes_to_use`) set to 64 GiB for Qwen, 256 GiB for Nemotron, and 64 GiB for Gemma. Secondary tiers were layered directly above this CPU allocation: local NVMe runs utilised a 7 TB Micron 7450 PRO NVMe SSD, while CephFS offload runs utilised a 3 TiB PersistentVolumeClaim backed by OpenShift Data Foundation 4.18 (configured as a 3-OSD cluster with dedicated VPC block backends). The CephFS Key-Value data pools were configured with a replication size of 1, storing a single copy per block.

The workload applied AIPerf’s AgentX MVP scenario alongside the SemiAnalysis Weka trace dataset (`semianalysisai/cc-traces-weka-062126`), replaying realistic multi-turn agent trajectories with a maximum context length of 131,072 tokens. The prompt distribution was heavily prefill-dominated, averaging 50,600 input tokens and 691 output tokens per request.

Throughput and Latency Results Across Model Classes

Performance gains varied significantly depending on whether the model’s Key-Value working set exceeded the capacity of the CPU offload tier.

* Nemotron-Ultra-253B: At an average input of 50K tokens, Nemotron’s cache demands exceeded the 256 GiB CPU tier retention capacity. Without offload, the model served 0.13 req/s with a mean time to first token of 59.7 seconds. CPU offload alone improved throughput to 0.44 req/s (3.3x). Adding a secondary tier nearly doubled that gain: local NVMe reached 0.76 req/s (5.7x) and CephFS reached 0.77 req/s (5.8x), while mean time to first token dropped to 9.6 seconds.
* Qwen3.6-35B-A3B: Because this model’s working set fit comfortably within its 64 GiB CPU allocation, CPU offload alone captured roughly 90% of the maximum achievable gain, rendering secondary tier additions largely incremental.
* Gemma 4 31B: High baseline graphics processing unit memory utilisation constrained the effectiveness of offload tiers, limiting overall throughput improvements.

Deployment Topologies: CPU, NVMe, and CephFS Trade-offs

Choosing the correct offloading topology requires balancing latency, node architecture, and resilience requirements.

* CPU-Only Topologies: Optimal for models whose working sets fit entirely within system DRAM. They avoid disk input-output overhead and complex storage provisioning.
* Local NVMe Topologies: Provide superior single-node cache locality with p99 latencies under 15 ms. They are ideal for dedicated, single-replica high-performance nodes handling massive context windows.
* CephFS Topologies: Introduce higher per-block latency (p99 ~451 ms) but provide shared, persistent storage across multiple inference replicas. In multi-replica tests, CephFS sustained high cache reuse and cross-replica hit rates as concurrency scaled, whereas CPU-only configurations fell back to expensive recomputation.

Step-by-Step Validation and Production Implementation

Platform engineers must execute a disciplined validation workflow before rolling out offloading tiers to production environments:

1. Characterise Storage Performance: Run `fio` benchmarks using libaio, direct input-output, and 16 jobs configured with your target model’s specific block size (calculated via the formula in Section 2). Record both p50 and p99 latencies.
2. Calculate Crossover Thresholds: Determine your model’s prefill cost per token ($t_{prefill}$) and apply the crossover formula $L_{crossover} = T_{FIO,p99} / t_{prefill}$ to verify that your target context distribution exceeds the break-even point.
3. Validate with Representative Traces: Deploy the vLLM offload connector in a staging cluster. Replay representative multi-turn inference traces (such as AIPerf AgentX scenarios) under expected concurrency levels.
4. Monitor Cache Hit Rates: Measure external cache hit rates, time to first token degradation, and p99 storage latencies in staging before promoting the configuration to production.

# Example: Execute storage characterisation fio test in staging
fio --rw=randrw --rwmixread=75 --bs=4M --iodepth=1 --numjobs=16 --direct=1 --ioengine=libaio --runtime=60 --name=kv-cache-test

Frequently asked questions

Why is sequential disk bandwidth insufficient for evaluating KV cache offloading?

Sequential throughput metrics measure sustained large-block transfer ceilings, which do not reflect real-world vLLM connector behaviour. vLLM reads and writes fixed 16-token Key-Value blocks whose byte sizes vary dynamically based on model architecture, head dimensions, and quantisation precision. Evaluating performance requires measuring per-operation random input-output latency, particularly p99 tail latency, under concurrent job loads that mimic actual serving conditions.

When should an administrator choose local NVMe over CephFS for a secondary tier?

An administrator should select local NVMe when deploying single-node inference instances that require the lowest possible secondary-tier latency (p99 under 15 ms) and where cross-replica cache sharing is unnecessary. CephFS should be chosen for multi-node deployments where shared cache pools, persistence across pod restarts, and cross-replica cache reuse outweigh higher file system protocol latencies.

How does working-set overflow affect CPU-only offload configurations?

When the active Key-Value cache working set exceeds the allocated CPU DRAM byte capacity, older blocks are evicted from the system memory tier. If incoming requests reference these evicted blocks, a cache miss occurs, forcing the graphics processing unit to recompute the entire prompt prefix from scratch. This recomputation negates offload benefits and degrades both time to first token and overall throughput.

What is the impact of minimal-replication (size 1) settings on CephFS cache resilience?

Configuring CephFS Key-Value data pools with a replication size of 1 stores only a single copy of each cached block across the cluster object storage devices. If an object storage device fails or undergoes maintenance, any Key-Value blocks residing on that device become immediately unavailable. The inference system handles this gracefully via cache misses, but the affected prefixes must be recomputed on the hardware accelerator, causing transient latency spikes.