VIDRAFT AX-Ray: Detecting Causal-Leakage Vulnerabilities in General-Purpose LLMs — Leaderboard Now on Hugging Face
TL;DR: VIDRAFT has released AX-Ray, an AI safety evaluation framework that diagnoses "causal leakage" — hidden, unintended causal pathways that can cause LLMs to bypass safety guardrails or behave unpredictably. The framework ships with 117 diagnostic items mapped to real-world legal and regulatory frameworks across multiple jurisdictions, and the leaderboard plus evaluation dataset are now publicly available on Hugging Face. Engineers working on LLM deployment, red-teaming, or compliance will want to take a look.
What it is
AX-Ray is a model safety diagnostics framework built by VIDRAFT (비드래프트), a deep-tech AI and scientific research company based in Seoul. Released on August 13, 2026, it focuses specifically on detecting causal leakage (인과누설) in general-purpose large language models — a class of vulnerability that has historically been discussed mostly in theory.
Key characteristics:
- 117 diagnostic items covering a broad spectrum of potential risk behaviors and model vulnerabilities
- Jurisdiction-aware mapping: each diagnostic item is cross-referenced against national laws and regulatory frameworks; coverage explicitly includes legal norms for Arabic-speaking regions as well as Islamic jurisprudence (Sharia-aligned norms), enabling culturally and legally contextualized safety assessment
- Leaderboard format: published on Hugging Face, enabling transparent, community-visible comparison across models
- Evaluation dataset: also publicly released alongside the leaderboard, so researchers can reproduce assessments or run their own models through the framework
- Positioned not as a performance benchmark (reasoning, coding, math) but as a safety and controllability diagnostic
VIDRAFT is also a resident company at Seoul AI Hub and an active developer of its own foundation model line (AETHER and its successor Darwin), though AX-Ray is designed to evaluate third-party general-purpose models, not just their own.
How it works
At a conceptual level, AX-Ray is built around the idea that a model can arrive at a correct-looking output via an incorrect — and potentially dangerous — causal path. Instead of measuring whether a model gets the right answer, it probes whether the model is relying on hidden informational shortcuts or unintended causal cues to reach its outputs.
The diagnostic pipeline works roughly as follows:
- Structured probing: The 117 diagnostic items act as targeted stimuli designed to surface specific risk-associated behaviors — things like safety rail circumvention, unexpected agentic actions, or system-level access attempts.
- Causal leakage signal detection: The framework identifies whether a model's response patterns suggest it is being influenced by latent causal dependencies not present in the intended reasoning chain.
- Regulatory mapping layer: Detected signals are cross-referenced against a taxonomy of legal and normative frameworks — national statutes, sector-specific regulations, and where applicable, religious-legal norms — to contextualize the severity and jurisdictional relevance of each finding.
This design reflects an important shift in the field: as LLMs get deployed in high-stakes domains (finance, healthcare, robotics, public services, defense), the question is no longer just "how smart is it?" but "can it act unpredictably in ways that violate law or social norms?"
Benchmarks & results
The article reports qualitative findings rather than aggregate numerical scores:
- AX-Ray evaluations detected causal leakage signals in two general-purpose public models, one of which is an NVIDIA general-purpose AI model
- These are described as confirmed causal leakage detections — the framework successfully diagnosed and reproduced the vulnerability in both cases
- This is framed as a significant result because detecting causal leakage in production-grade general-purpose LLMs had previously remained an open, hard problem — prior work largely confined the discussion to theoretical risk or narrow controlled settings
No specific numerical scores, pass/fail rates, or per-item breakdowns were reported in the source article. The full leaderboard on Hugging Face is the authoritative location for quantitative results.
How to try it
The AX-Ray leaderboard and evaluation dataset are publicly available on Hugging Face. The source confirms this is an open release as of August 13, 2026. You can browse the leaderboard or pull the dataset directly:
# Browse the leaderboard on Hugging Face (search for "AX-Ray" or "VIDRAFT")
huggingface-cli search datasets vidraft
Specific dataset repository names and model submission instructions were not detailed in the source article. Check the VIDRAFT organization page on Hugging Face for exact repository slugs and submission guidelines.
No GitHub repository URL, OpenAI-compatible API endpoint, or pip install package was mentioned in the source at this time.
FAQ
Q: What exactly is "causal leakage" and why is it dangerous in deployed LLMs?
A: Causal leakage refers to a model being influenced by hidden or unintended causal signals — rather than the explicit reasoning chain you expect — when generating outputs. In practice, this can manifest as an LLM bypassing safety guardrails, performing unexpected system-level actions, or behaving in ways that are difficult to anticipate or audit. It's particularly concerning in agentic settings (autonomous agents, robotics, financial automation) where the model has real-world action capability.
Q: Can I submit my own model to the AX-Ray leaderboard?
A: The leaderboard is publicly available on Hugging Face, which suggests community submission is at least structurally possible, but the source article does not describe a formal submission process. Check the VIDRAFT Hugging Face organization page for current submission instructions.
Originally reported by 전자신문 (2026-08-14) — source article.
