AX-RAY: VIDRAFT's Open AI Safety Diagnostic Leaderboard & Dataset Now on Hugging Face
TL;DR: VIDRAFT has publicly released AX-RAY, a 117-item AI safety diagnostic framework, along with an open leaderboard and evaluation dataset on Hugging Face. The system detects "Causal Leakage" — a subtle failure mode where LLMs are influenced by hidden or unintended causal cues rather than legitimate reasoning paths. Two publicly available general-purpose AI models showed anomalous signals on the Causal Leakage dimension, including one from NVIDIA.
What it is
AX-RAY is VIDRAFT's AI safety evaluation framework designed to diagnose latent risk behaviors and safety vulnerabilities in large language models (LLMs) and AI agents. It is structured around 117 diagnostic evaluation items and goes beyond capability benchmarking to assess a model's potential for norm-violating or unsafe behavior.
Key distinguishing features of AX-RAY:
- Causal Leakage detection: Identifies signals indicating a model's decisions are influenced by hidden information or unintended causal shortcuts — rather than through correct reasoning chains.
- Jurisdiction-aware evaluation: Safety assessments are mapped to national laws and regulatory frameworks. For example, evaluations targeting Arabic-speaking regions incorporate not only local statutory law but also religious and social normative systems (e.g., Sharia-based frameworks).
- Open leaderboard format: AX-RAY operates as a publicly visible leaderboard, enabling the community to track how different models compare across safety dimensions over time.
- Dataset publicly available: The evaluation dataset powering AX-RAY is published on Hugging Face, making it accessible for researchers and developers who want to run their own evaluations or audit models independently.
The intended scope is broad: finance, healthcare, robotics, and public services — domains where LLM misbehavior against legal or normative standards carries real-world consequences.
How it works
At a conceptual level, AX-RAY probes models with structured evaluation items designed to surface latent behavioral patterns that standard capability benchmarks miss.
Causal Leakage, the core diagnostic signal in this release, refers to a model being steered by spurious or hidden causal cues embedded in its inputs or context — rather than reasoning transparently from stated evidence. Think of it as the AI equivalent of a human expert whose judgment is unconsciously biased by irrelevant contextual signals they can't articulate.
The framework's diagnostic approach works roughly as follows:
- Structured prompting across 117 safety-relevant categories exposes models to scenarios where Causal Leakage and other risk behaviors might manifest.
- Anomaly detection on model outputs identifies response patterns that deviate from expected safe behavior on those items.
- Regulatory mapping cross-references detected risks against applicable legal and normative standards for a given jurisdiction or cultural context.
Importantly, VIDRAFT is explicit that detecting an anomalous signal in the Causal Leakage dimension does not mean the model has caused harm or compromised any system. It indicates a potential vulnerability that warrants further investigation — closer to a security scanner flagging a suspicious pattern than a confirmed exploit.
Benchmarks & results
The source article does not publish quantitative scores or percentile rankings, but provides these qualitative findings:
- Two publicly released general-purpose AI models showed abnormal signals specifically on the Causal Leakage evaluation dimension.
- One of the two flagged models is from NVIDIA.
- The anomalous signals represent potential safety vulnerabilities, not confirmed dangerous behaviors or system compromises.
VIDRAFT plans to expand both the number of models evaluated and the scope of the diagnostic dataset over time as the leaderboard matures.
How to try it
The AX-RAY leaderboard and evaluation dataset are publicly available on Hugging Face. According to VIDRAFT's announcement, developers can access them directly via the Hugging Face platform.
To browse or download the dataset using the Hugging Face CLI:
pip install huggingface_hub
huggingface-cli login
# Then search for VIDRAFT's AX-RAY dataset on huggingface.co/vidraft
Note: Specific dataset repository slugs and leaderboard URLs were not published in the source article. Visit huggingface.co and search for VIDRAFT or AX-RAY to locate the official dataset and leaderboard space. No API endpoint, model weights, or SDK beyond the dataset have been announced as part of this release.
FAQ
Q: Is Causal Leakage the same as prompt injection or jailbreaking?
A: Not exactly. Prompt injection involves deliberate adversarial manipulation of a model's input. Causal Leakage is a subtler, often unintentional phenomenon — the model's internal reasoning is swayed by hidden or spurious causal signals in its context, even without an adversary deliberately engineering them. It's more of a structural reasoning flaw than an attack vector, though it could potentially be exploited.
Q: Does the leaderboard only evaluate safety, or does it also cover capability?
A: Based on the source, AX-RAY is exclusively a safety diagnostic framework. It is not designed to benchmark performance on reasoning, coding, or knowledge tasks. Its 117 evaluation items specifically target latent risk behaviors, potential norm violations, and regulatory non-compliance — making it complementary to, not a replacement for, capability benchmarks like MMLU or HumanEval.
Q: Why does AX-RAY include jurisdiction-specific and cultural normative frameworks?
A: AI safety is not culturally neutral. A model deployed in a financial advisory context in one country may face entirely different legal obligations than the same model deployed in a healthcare context elsewhere. By mapping diagnostic results to local laws and — in some cases — religious or social normative systems, AX-RAY aims to make safety evaluations actionable for teams building region-specific or regulated-industry applications.
Q: Does being flagged on AX-RAY mean a model is unsafe to use?
A: No. VIDRAFT is clear that a flagged Causal Leakage signal represents a potential vulnerability identified in evaluation scenarios — not evidence that the model has produced harmful outputs or security breaches in production. Teams should treat it as a signal for deeper investigation in their specific deployment context.
Originally reported by IT조선 (2026-08-18) — source article.
