Memory tiering and KV-cache offloading

When a model’s KV cache outgrows the capacity of the accelerator’s HBM — which happens at long context, high concurrency, or both — the serving system has two choices: throw cache away and recompute it later, or move it somewhere cheaper and larger and fetch it back when needed. Memory tiering is the strategy of arranging a hierarchy of memories by speed and cost; KV-cache offloading is the act of pushing cache blocks down that hierarchy to relieve HBM pressure. This unit builds directly on the KV-cache and HBM foundations: the cache’s linear growth with sequence length and batch size is the demand, and HBM’s fixed per-device capacity is the ceiling that forces the question.

The memory hierarchy, tier by tier

The tiers trade bandwidth and latency against capacity and cost per gigabyte, and the gaps between them are best understood as orders of magnitude rather than precise numbers (the exact figures below are approximate and come partly from secondary and vendor material). This is the same bandwidth-versus-capacity-versus-cost tension that the GDDR-vs-HBM fork draws within a single device’s memory technology, now extended across devices into a vertical hierarchy: where that unit contrasts wide-slow-stacked HBM against narrow-fast GDDR, tiering stacks HBM, cheaper-but-slower host DRAM, and cheap-but-far-slower NVMe on top of one another and moves data between them.

• HBM (on-package) — the fast tier that holds model weights and the hot KV cache. Aggregate device bandwidth runs in the terabytes per second (roughly 3.35 TB/s on an H100 up to ~8 TB/s on a B200), with capacity fixed at 80–192 GB per accelerator. This is where decode must stream from: decode is memory-bandwidth-bound, sitting deep in the memory-bound region of the roofline model (arXiv 2402.16363), so any block the GPU must attend to this step has to be resident in HBM at full bandwidth.
• CPU / host DRAM — the primary offload target. Capacity is far larger (order of ~1 TB across an 8-GPU node) and cost per GB far lower, but the GPU reaches it across an interconnect, not on-package. Over PCIe the effective bandwidth is roughly an order of magnitude below HBM (PCIe Gen5 ×16 is about 64 GB/s nominal; Gen4 ×16 delivers ~25 GB/s in practice), and small-transfer latency is microseconds rather than the sub-microsecond HBM access.
• Unified-memory interconnect (NVLink-C2C) — on tightly coupled parts such as NVIDIA’s Grace Hopper, a coherent CPU–GPU link (NVLink-C2C, ~900 GB/s bidirectional, roughly an order of magnitude beyond PCIe Gen5) narrows the gap to host memory dramatically and makes DRAM offload far cheaper than on a PCIe-attached system (NVIDIA Grace Hopper architecture). This is why the economics of offloading depend heavily on the interconnect, not just the tiers.
• Local NVMe / SSD — the cold tier, for cache that may be reused but is not needed soon. Capacity is effectively unbounded and cheap, but bandwidth is single-digit to low-tens of GB/s and latency is microseconds to sub-milliseconds — slow enough that, as several research deployments report, restoring a large cold cache from NVMe can rival or exceed simply recomputing it.
• CXL-attached memory — a proposed middle tier (memory-semantic, cheaper per GB than HBM, latency of a few hundred nanoseconds) that in principle sits between DRAM and SSD. As of 2026 it remains largely research-stage for LLM serving; no major serving system has deployed CXL KV tiering, so it is noted here as a direction, not a shipping tier.

A common policy assigns blocks by recency: hot blocks in active decode stay in HBM; warm blocks (completed turns, multi-turn history) move to CPU DRAM; cold blocks (historical prefixes) spill to NVMe. This is the concrete shape of tiering.

Offloading is not eviction

The distinction is the crux of the whole topic. Eviction discards KV blocks outright; if they are needed again the model must recompute them from the tokens, paying a full prefill cost. Offloading preserves the blocks in a slower tier so they can be copied back on a later hit, trading a data-movement cost for the avoided recomputation. TensorRT-LLM makes the choice explicit: reusable blocks can be offloaded to host memory as an alternative to eviction, extending their lifetime and improving cache hit rates (NVIDIA TensorRT-LLM KV cache reuse). The trade is only worth it when the round-trip to fetch a block back is cheaper than recomputing it — which, on the slower tiers, is not always true.

How the mechanics work

• Granularity. Modern serving allocates the cache in fixed-size blocks rather than contiguous per-request buffers — vLLM’s PagedAttention uses blocks of a small number of tokens (16 by default) and identifies reusable prefixes by content hash (arXiv 2309.06180). Offloading operates at this block granularity: whole blocks move between tiers.
• Trigger. The usual trigger is HBM pressure — when the free GPU block pool falls below a threshold, low-priority blocks are moved out while high-priority blocks (e.g. a shared system prompt) stay resident. TensorRT-LLM exposes this as per-range priorities with LRU tie-breaking (NVIDIA TensorRT-LLM KV cache reuse).
• Transfer. Offload and recall use asynchronous DMA copies (e.g. cudaMemcpyAsync) that overlap with ongoing compute, so on a cache hit the promotion of a block from DRAM back to HBM is largely hidden from the user (vLLM documentation). Whether it stays hidden depends on whether there is enough compute to overlap the transfer against — which is not guaranteed.
• Prefetching. DeepSpeed/ZeRO-Inference overlaps the fetch of the next layer’s offloaded data with the current layer’s compute to hide transfer latency; the overlap helps materially for NVMe offload but much less for CPU offload, because a single decode layer does not take long enough to hide a host-memory fetch (DeepSpeed ZeRO-Inference). Latency hiding is thus tier-dependent, not free.

The framework landscape

Different systems make different bets, which is itself informative about the design space:

• vLLM — extends prefix caching by asynchronously offloading completed KV blocks to pinned CPU memory as they are produced, with a CPU-primary tier that has direct GPU access and optional secondary tiers that do not (vLLM documentation). Separately, an open vLLM issue (#33864) reports, as of this writing, that blocks formed during decode (as opposed to prefill) are not offloaded because reuse keys cover only already-computed tokens — an implementation limitation that may since have been resolved, not stable documented behavior.
• TensorRT-LLM — priority-based LRU with offload-as-an-alternative-to-eviction to host memory, reported to improve reuse/hit rates, with the offload cost small on x86+Hopper and near-negligible on Grace-Hopper’s coherent link (NVIDIA TensorRT-LLM KV cache reuse).
• DeepSpeed / ZeRO-Inference — offloads to CPU and optionally NVMe with layer-wise prefetching; a throughput-oriented design for running very large models on constrained GPUs (DeepSpeed ZeRO-Inference).
• FlexGen — computes a static placement across GPU/CPU/disk with a linear-programming solver before execution, then runs a schedule that maximizes weight reuse across a large effective batch. It reports up to roughly 100× higher generation throughput than prior offloading systems for OPT-175B on a single commodity GPU — but this is an explicitly throughput-maximizing, latency-insensitive regime (batch synthesis), not interactive serving (arXiv 2303.06865).
• LMCache — runs as a standalone daemon managing a persistent, tiered KV cache (GPU → CPU DRAM → local disk → remote backends), enabling reuse across requests, sessions, and even separate engine instances (LMCache documentation). Its CacheBlend line of work extends reuse to non-prefix positions by selectively recomputing a small fraction of high-deviation tokens; the reported speedups are vendor-published and treated here as indicative, not settled.

The taxonomy has a clear axis: vLLM and TensorRT-LLM optimize request- and sequence-level reuse for latency-sensitive serving; FlexGen and ZeRO-Inference optimize aggregate throughput for large-model batch work; LMCache adds an infrastructure layer for cross-instance reuse.

What offloading actually buys — and what it costs

Offloading is fundamentally a throughput and capacity optimization, not a latency one. Its wins concentrate where HBM capacity would otherwise cap the batch size or force prefill recomputation — high concurrency, long context, and strong prefix reuse — and its costs land as added latency on the constrained data path.

• Throughput. Where KV pressure caps batch size, offloading can restore a large fraction of the batch and therefore of decode throughput, provided the interconnect is not saturated. The high multipliers quoted for offloading are regime-specific: FlexGen’s ~100× is a single-GPU, batch-maximizing figure (arXiv 2303.06865), and vendor serving benchmarks reporting several-fold gains assume favorable hit rates and batch sizes.
• Time-to-first-token (TTFT). The effect splits sharply on cache hit vs miss. On a hit — a previously seen prefix already offloaded — restoring the cache is far cheaper than recomputing it, so TTFT improves substantially (vendor benchmarks report multi-fold gains, e.g. reusing a long offloaded prefix instead of a multi-second prefill). On a miss, offloading adds essentially nothing to the initial prefill, so TTFT is unchanged at best. Offloading rewards workloads that reuse prefixes; it does nothing for cold, unique prompts.
• When it pays off vs. stalls. It pays off in high-batch, long-context, high-reuse pipelines. It stalls in latency-sensitive interactive use, small-batch inference, and PCIe-saturated multi-tenant clusters, where contention on the offload path can push tail latency up sharply. A reasonable operating rule from the field: interactive products targeting low TTFT should start with in-HBM prefix caching and add offloading only under genuine memory pressure. Note that prefix caching (reuse without leaving HBM) is orthogonal to and often more powerful for TTFT than offloading, and the two should not be conflated.

A genuine, unresolved tension in the evidence. Vendor benchmarks (vLLM, storage vendors) report large throughput and TTFT gains, while research papers on realistic multi-tier deployments report that SSD-tier offloading often degrades end-to-end latency versus selective recomputation, and that CPU-DRAM offload amortizes only at high batch where PCIe contention is already severe. The honest reconciliation is that the gains are real but narrowly scoped to high-concurrency, long-context, high-reuse serving; general-purpose inference sees modest or even negative returns. This unit deliberately does not adopt the headline vendor multipliers as its own claim.

Cross-pillar: what maturing offloading means for HBM demand

The following is analysis, hedged and stated as such.

It is tempting to read offloading as a way to reduce the need for scarce, premium HBM. The mechanics do not support that reading. Offloading shifts and multiplies the total memory footprint rather than shrinking it: it keeps hot KV in HBM and spills warm/cold blocks into DRAM and NVMe, so total system memory per node tends to rise, not fall, and the technique adds interconnect, storage, and software complexity. It may let a given accelerator serve longer contexts or more concurrent users — relieving peak per-GPU HBM capacity pressure — without lowering aggregate memory demand. Given that HBM remains premium and supply-gated by advanced packaging (the character this base’s HBM overview and HBM pricing unit already track), offloading is best understood as a coping strategy under HBM scarcity — one that redistributes demand toward DRAM, NVMe, and faster interconnects — rather than a cost reduction. This is analysis of a technical dynamic, not a market forecast or any form of advice.

Confidence 0.55: the structural core is solid and internally consistent with this base’s KV-cache and HBM units — the tier ordering, the offload-vs-eviction distinction, the block-based mechanics, the framework taxonomy, and the throughput-not-latency framing are corroborated across primary framework documentation and stable academic anchors (FlexGen, PagedAttention, ZeRO-Inference, the roofline survey). It is held below the KV-cache unit’s 0.6, and well below the HBM unit, because most quantitative performance claims (throughput and TTFT multipliers, tier bandwidth/latency figures, break-even concurrency) come from vendor benchmarks and secondary or unverified sources that this pipeline did not independently confirm, and because the field genuinely disagrees on the magnitude of the gains. Several arithmetic and figure claims were deliberately dropped to qualitative statements rather than pinned to weak or non-resolving citations. Confidence should rise as the interconnects unit (NVLink/PCIe/CXL) and primary benchmarks pin these numbers down.