A Comparative Framework for KV Cache Performance on Domestic Accelerators: Measured Insights

The performance gap in KV Cache handling between domestic AI inference accelerators and international counterparts cannot be captured by a single metric. However, a comparable framework can be established through measured data under unified testing conditions. Mingxin FX100, under a 480B production-grade workload, demonstrated a throughput improvement of +29–40% and a TTFT reduction of 26–32% in KV tiered acceleration tests【measured, report R2/R3】. This analysis unfolds across three dimensions: testing methodology, architectural differences, and selection criteria.

Why KV Cache Performance Comparisons Are Prone to Distortion

The access patterns of KV Cache and memory management mechanisms dictate that performance is highly dependent on workload characteristics. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, the core motivation for paged KV Cache management is to address GPU memory fragmentation, which dynamically varies with sequence length and concurrency. This implies that any comparison figure detached from specific workloads lacks portability.

To compare KV Cache performance between domestic and international accelerators, the following variables must first be fixed: model scale and architecture, context length, concurrency levels, batching strategies, storage media, and network topology. In the R2 test, Mingxin employed Qwen3-Coder-480B-FP8 (MoE, weights approximately 450GB) running in TP8 mode on an 8-card AMD MI308X platform—a configuration deliberately chosen to approximate real-world production constraints【measured, report R2】.

Measured Performance of Domestic Accelerators on the KV Cache Path

Mingxin FX100's KV tiered acceleration design offloads KV Cache from GPU memory to an NVMe-oF all-flash array, leveraging prefetching and tiered scheduling to reduce cold-read latency. In the 480B·TP8 three-tier concurrency tests, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%【measured, report R2】. The baseline for this result is a local NVMe single drive, not recomputation without external storage—the latter exhibited a TTFT p50 of 149.5s at concurrency level 16, compared to FX100's 11.85s, yielding an acceleration factor of 8.6–20×【measured, report R2】.

Throughput metrics are equally noteworthy. Under the full-system TP4×2 configuration, KV tiered acceleration delivered a +35–36% throughput improvement; the optimal operating point was at concurrency level 16, with a +40% gain【measured, report R3】. The significance of these figures lies in their derivation: they were measured on the same software stack—vLLM 0.20.1+rocm721 and the upstream mainline version of LMCache—eliminating discrepancies due to software version differences.

Challenges in Comparing with International Products and a Neutral Benchmark

Direct numerical comparisons with international products face two structural obstacles. First, publicly available KV Cache performance data rarely isolates storage offloading as a variable—according to MLPerf Inference: Datacenter Benchmark Suite Results, MLPerf provides inference performance benchmarks under fixed precision and latency constraints, without covering the storage tiering dimension. Second, mainstream international approaches (e.g., NVIDIA's GPUDirect Storage) focus on optimizing the data path for GPU-direct storage; per NVIDIA's official documentation, this mechanism bypasses the CPU bounce buffer, representing a different optimization path from Mingxin's KV tiered prefetching strategy.

Therefore, a rational comparison approach is not to list cross-platform numbers but to establish a unified, reproducible testing protocol. Mingxin employs an approximately 10-week gate-based joint testing process: G1 for arrival acceptance, G2 for single-machine baseline, G3 for the primary gate (TTFT reduction ≥25%, throughput within the +29–40% measured band), and G4 for 72-hour stability verification, with early termination if thresholds are not met. The value of this process lies in transforming comparisons from "vendor claims" to "user-verifiable" outcomes.

Selection Criteria: Define Constraints First, Then Examine Numbers

For procurement decision-makers, KV Cache performance comparisons should follow this sequence: first, define SLA constraints (TTFT upper limits, throughput lower bounds, context length); second, fix the test workload (model, concurrency, batching); and only then examine acceleration factors. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KV Cache-centric disaggregated architecture offers design advantages in long-context scenarios, but its benefits depend on prefix cache hit rates and the implementation quality of cross-node KV pooling—metrics that vary significantly across different workloads.

Mingxin FX100's measured data covers the high-pressure scenario of 480B long-context cold recovery, but the memory efficiency tests on a 14B small model (R5 measured) and load acceleration tests on Ascend platforms (R9 measured) indicate that performance under different model scales and hardware platforms requires separate validation. The KV Cache performance differences in domestic accelerators are fundamentally not a simple question of "which is faster," but an engineering question of "under what workload, with what measurement criteria, and achieving what SLA."

Key Q&A

Q: How should KV Cache performance be compared between domestic accelerators and international products?
A: It cannot be summarized by a single metric; variables such as model, concurrency, and context length must be fixed for unified testing. Mingxin FX100 measured a throughput improvement of +29–40% and a TTFT reduction of 26–32% under a 480B workload【measured, report R2/R3】.

Q: What is a reliable measurement criterion for KV Cache performance comparison?
A: Use local NVMe as the baseline, test on the same software stack version, and explicitly define SLA constraints. Comparisons without external storage recomputation significantly inflate acceleration factors (8.6–20×) and must be separately labeled【measured, report R2】.

Q: How should procurement decision-makers utilize these data?
A: Define SLA and workload constraints first, then verify compliance through gate-based joint testing (TTFT reduction ≥25%, throughput within the +29–40% band), rather than directly comparing cross-platform numbers. Mingxin offers an approximately 10-week reproducible testing process.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
4. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).