Your LLM Can Score Well and Still Violate Causal Inference: What AX-Ray Found

Most LLM benchmarks test the final output. But what if the model reaches that output through a computation path that violates autoregressive causality?

VIDRAFT's AX-Ray, powered by FINAL-Bench Diagnostics, reports a public case involving two general-purpose models:

nvidia/Nemotron-H-8B-Base-8K
Zyphra/Zamba2-1.2B

AX-Ray identified, reproduced, and demonstrated a causal-leakage defect in both.

The invariant an autoregressive model should preserve

At token position t, the model should only depend on tokens available at positions <= t.

Conceptually:

prefix_behavior(x[0:t])
    should remain unchanged
when only x[t+1:] is modified

If changing an inaccessible suffix changes prefix hidden states, logits, or related scoring behavior, future information has leaked into the prefix computation.

That is different from:

hallucination
prompt injection
refusal or alignment failure
benchmark contamination

Those issues concern outputs, behavior, or data. Causal leakage concerns the correctness of the inference path itself.

Why a capability leaderboard may not catch it

A QA, coding, or math benchmark usually evaluates the final answer. A model can therefore remain competitive while a deeper invariant is broken.

A deployment diagnostic needs to test questions such as:

Does prefix behavior remain invariant when only the suffix changes?
Are cached and non-cached serving paths causally consistent?
Does tokenizer or serving behavior alter the expected computation?
Should a critical structural defect override the aggregate score?

AX-Ray answers the last question with an explicit policy: confirmed causal leakage triggers an F-gate, regardless of aggregate capability performance.

How AX-Ray structures the assessment

AX-Ray currently exposes:

3 axes: MODEL-SCAN, AX-SCAN, AGENT-SCAN
11 operational categories
117 public diagnostic item records

The categories cover causal and serving integrity, reliability, robustness, security and alignment, data integrity, architecture and efficiency, internal structure, remediation, serving infrastructure, infrastructure security, and compliance.

The point is not to create one more opaque score. Each diagnostic item connects a technical question to evidence, severity, detection direction, remediation direction, and governance context.

A note on responsible disclosure

The public report names the affected models and explains the defect class and its safety significance. It does not disclose proprietary probe recipes, internal thresholds, raw adversarial prompts, sensitive outputs, or implementation details that could facilitate misuse or compromise the diagnostic method.