VIDRAFT's vidraft-darwin Sets Top Marks on Google & Hugging Face's Fast Gemma Challenge: 510.58 TPS on a Single A10G
TL;DR: VIDRAFT (비드래프트), a Korean Pre-AGI AI startup, has achieved the highest verified score in the Verification category of the The Fast Gemma Challenge — a joint benchmark hosted by Google and Hugging Face — with its model vidraft-darwin hitting 510.58 tokens per second (TPS) and a perplexity (PPL) of 2.39 on a single NVIDIA A10G GPU. These are publicly validated figures, not internal claims, making this a meaningful data point for engineers evaluating efficient inference at edge-class hardware constraints.
What it is
vidraft-darwin is VIDRAFT's submission to The Fast Gemma Challenge, a community inference benchmark co-organized by Google and Hugging Face designed to push throughput and language modeling quality on the Gemma model family.
Key verified facts from the source:
-
Model name:
vidraft-darwin - Challenge: The Fast Gemma Challenge — Verification category
- Organizers: Google & Hugging Face
- Achievement: Top recorded score in the Verification category as of the benchmark's public leaderboard
- Throughput: 510.58 tokens per second (TPS)
- Perplexity: 2.39 PPL
- Hardware constraint: Single NVIDIA A10G GPU
- Coverage: Featured in the Engineering & Research section of TLDR AI, a developer newsletter with over 1.1 million subscribers, on 2026-08-04
The A10G is a mid-tier data center GPU (24 GB VRAM), meaning this result was not achieved on a high-end multi-GPU cluster — which is precisely what makes it notable for practitioners working within realistic deployment budgets.
How it works
The source does not disclose VIDRAFT's internal implementation details, and we will not speculate. What can be said at a conceptual level based on the nature of the benchmark:
- The Fast Gemma Challenge evaluates submissions on inference throughput (how many tokens per second a model can generate) alongside language modeling quality (perplexity — lower is better).
- Achieving both high TPS and low PPL simultaneously is the core tension: aggressive quantization or batching strategies that maximize throughput often degrade perplexity, so optimizing both metrics together is non-trivial.
- Running competitively on a single A10G suggests the approach prioritizes memory efficiency and latency-aware inference — relevant for teams deploying on single-GPU cloud instances or on-premise workstations rather than large multi-GPU nodes.
- The Verification category specifically implies results were independently validated by the benchmark organizers, not self-reported — a meaningful distinction for reproducibility.
Benchmarks & results
All figures below are drawn directly from the source. No numbers have been inferred or extrapolated.
| Metric | Value | Context |
|---|---|---|
| Throughput | 510.58 TPS | Tokens per second, inference |
| Perplexity (PPL) | 2.39 | Lower = better language modeling quality |
| Hardware | Single NVIDIA A10G GPU | No multi-GPU scaling |
| Leaderboard position | #1 in Verification category | As of publication date |
A PPL of 2.39 is competitive for a model optimized for speed, and 510+ TPS on a single A10G is a throughput figure that has direct practical implications — for example, serving multiple concurrent users on a single GPU instance without scaling out.
How to try it
The source article does not include explicit public links to a Hugging Face repository, GitHub codebase, or API endpoint for vidraft-darwin at the time of writing.
Given that the submission was made to an official Google & Hugging Face challenge, the leaderboard and potentially the model weights may be publicly accessible through:
- Hugging Face leaderboard: Check huggingface.co and search for The Fast Gemma Challenge leaderboard
-
VIDRAFT's Hugging Face organization: Search
vidrafton Hugging Face Hub for any public model releases - VIDRAFT's official channels: Monitor announcements for API or weight releases
⚠️ We will not provide invented endpoints, model slugs, or shell commands. If and when VIDRAFT publishes weights or an OpenAI-compatible API endpoint publicly, standard access patterns (e.g.,
huggingface-cli download,curlagainst an OpenAI-compatible base URL) would apply.
FAQ
Q: What exactly is The Fast Gemma Challenge, and who validates the results?
A: It is a community inference benchmark co-organized by Google and Hugging Face focused on maximizing throughput and maintaining language quality on the Gemma model family. The Verification category means submissions undergo independent validation by the organizers — results are not solely self-reported by participants.
Q: Is a PPL of 2.39 good for a high-throughput model, and should I trust it as a proxy for downstream task quality?
A: Perplexity measures how well a model predicts a held-out text distribution — lower is better. A PPL of 2.39 is a strong result in the context of a throughput-optimized submission. That said, PPL is a proxy metric; it correlates with general language modeling quality but does not directly predict performance on specific downstream tasks (e.g., coding, reasoning, instruction-following). Treat it as one signal among many when evaluating fit for your use case.
Q: Can I reproduce this result on my own A10G instance?
A: The hardware constraint (single NVIDIA A10G) is public, but the exact configuration — inference engine, quantization scheme, batching strategy — has not been disclosed in the source. Watch VIDRAFT's public channels for a technical write-up or reproducibility guide.
Originally reported by TLDR AI (미국) (2026-08-04) — source article.
