The Same LLM, the Same Weights, the Same Prompt — Different Hardware Gives Different Answers

Description: "Introducing Ruitong: the first public accuracy delta table for LLMs across CUDA, Ascend, and AMD. Same model, same weights, same prompt — different hardware gives different answers. Nobody publishes this number. We do."
tags: [llm, inference, hardware, benchmarks, gpu]
canonical_url: https://ruitong.io

cover_image: https://ruitong.io/matrix-banner.jpg

Everyone knows that moving an LLM across hardware changes the cost and the latency. What nobody talks about is that it also changes the answers.

The same model, the same weights, the same prompt, the same seed — different accelerators produce different output. This isn't a bug. Floating-point arithmetic is not associative, so different kernels reduce in a different order and the logprobs shift. It's expected, unavoidable, and almost never measured.

Until now.

The Problem

If you move inference from NVIDIA CUDA to Huawei Ascend, AMD MI300X, or even change precision (fp16 → bf16) on the same GPU, the output distribution shifts. The industry has good benchmarks for throughput and latency. Nobody publishes a reference number for accuracy equivalence.

Huawei's msprobe tells you operator-level tensor diffs. vllm-ascend's CI accepts 5% drift against a hardcoded YAML value with no GPU baseline. CAICT's national adaptation programme scores five dimensions — accuracy equivalence is not one of them. MLCommons publishes 99%/99.9%-of-FP32 for a handful of benchmark models.

Every team migrating inference independently discovers the same drift. And independently decides what tolerance is acceptable. That's a lot of duplicated effort.

What Ruitong Is

Ruitong is a cross-accelerator equivalence reference — a dated, versioned, reproducible accuracy delta table that answers one question: how much does the output actually change?

The three rows that matter most

1. Same hardware, different precision (fp16 vs bf16) — establishes the noise floor. Everything else is read against it.
2. Different vendor, same precision (NVIDIA vs Ascend/AMD) — the actual question a migrating customer has.
3. Same hardware, different batch size — published research shows bf16 accuracy varying up to 9% from GPU count and batch size alone.

Key Finding So Far

On A100, across fp16 vs bf16 — same hardware, same vendor, different precision — 18 out of 61 prompts produced different output text. All calibrated equivalence gates still passed.

This is the noise floor. The baseline against which every cross-vendor delta must be evaluated.

Beyond Token-Level Metrics

We run calibrated equivalence gates — end-to-end tests that check whether the candidate output is functionally equivalent to the reference (same classification, same choice, same numeric output within tolerance). Our finding: even when per-token metrics show drift, calibrated gates often pass. This has implications for how inference benchmarks should be designed.

The Methodology

• Models: Qwen3-8B (primary), Llama-3.1-8B, DeepSeek variants
• Hardware: NVIDIA A100 80GB (reference), Ascend 910B 64GB, Ascend 910C 128GB, AMD MI300X
• Precisions: fp16, bf16, W8A8
• Metrics: top-k max Δlogprob, prob mass shift, top-1 agreement, top-5 set agreement, calibrated equivalence gates
• Tooling: Open-source Python pipeline, reproducible via command line

Why This Matters

Every company migrating inference from CUDA to Ascend (or any accelerator) needs to answer one question: "Does the model still work correctly?"

Without a published reference, every team independently discovers the same drift — and independently decides what tolerance is acceptable. Ruitong publishes the reference so the industry has a single number to cite, argue with, and improve against.

Current Status

Contribute

The method and tooling are open. If you run inference on different hardware and want your bench to have a comparable number, this is the reference. Pull requests, issue reports, and hardware contributions welcome.

瑞通 Ruitong — auspicious flow between ecosystems.

ruitong.io · GitHub