Three months into running aiappdex.com, I found that roughly 10% of the model entries in the database still had fallback template content. You could spot them instantly: "X is an open-source image-text-to-video model available on HuggingFace. Details are sourced from the public model registry." Every sentence identical except the model name and pipeline tag.
This is what happens when you launch a directory that ingests thousands of models fast. Claude Haiku 4.5 generates proper content for each entry, but if the API key isn't set, or a model hit a rate limit, or an ETL run timed out mid-loop — the fallback activates. The fallback is intentional and safe. The problem is when it persists.
Here's how I built a nightly upgrade pass that finds these entries and replaces them with specific, grounded content.
Why template residue is worth fixing
A model entry that says "X is an open-source fill-mask model available on HuggingFace" is not useless — it's at least accurate. But it doesn't answer the question a developer actually arrives with: should I use this for my task, or not? The three-tier content quality ladder I described earlier puts template residue in tier 1 (machine-safe) rather than tier 3 (decision-ready). Tier 1 pages are indexed, but they don't convert and they don't hold a reader.
The more concrete concern: once AdSense review is in the picture, a page that says the same five words in 300 entries is a "low-value content" signal. Fixing template residue is both a content quality issue and an indexing hygiene issue.
Detecting the entries
The ETL marks every entry with a model_used field in the model_content table. When Claude generates content, model_used gets the model identifier (e.g. claude-haiku-4-5). When the fallback runs, model_used is fallback-template. When a seeded JSON import runs without Claude, it's seeded-from-json. This provenance field is part of the quality contract approach I use across both the article pipeline and the model directory — knowing what generated each piece of content is what lets you target upgrades precisely.
The detection query is straightforward:
SELECT m.id, m.name, m.pipeline_tag, m.tags
FROM models m
LEFT JOIN model_content c ON c.model_id = m.id
WHERE c.model_id IS NULL
OR c.model_used IN ('fallback-template', 'seeded-from-json')
ORDER BY m.downloads DESC
LIMIT ?
Two things about this query: the LEFT JOIN with c.model_id IS NULL catches models that never got any content row at all, which happens when new models are ingested before the content generation step runs. The ORDER BY m.downloads DESC prioritizes high-traffic entries — if I can only upgrade 50 models tonight due to rate limits, I want those 50 to be the ones with real search demand.
This detection pattern is the same one I described in detecting template residue in AI-generated directory content, but here I'm running it to actually upgrade rather than just audit.
The system prompt that makes the difference
The output quality lives almost entirely in the system prompt. The JSON extraction pattern below runs the same way across all three ETLs. Here's the system prompt itself:
You are an expert AI/ML engineer writing concise, factual directory entries
for open-source AI models. For each model, produce:
1. A 2-3 sentence technical summary (no hype)
2. 3-5 concrete use cases
3. 3-5 pros (technical strengths)
4. 3-5 cons (honest trade-offs)
Output ONLY a JSON object with keys: summary (string), use_cases (string[]),
pros (string[]), cons (string[]).
No markdown, no prose outside the JSON.
Focus on facts a developer evaluating the model would want to know.
The "no hype" constraint is doing a lot of work. Without it, Claude tends to write summaries like "a powerful and versatile model for a wide range of tasks." With it, summaries are specific: architecture details, quantization method, license constraints, format compatibility.
The user prompt sends model name, pipeline tag, and tags — that's it. No fetching the model card live (too slow and rate-limited). Claude infers from its training what it knows about these models. For well-known models like bert-base-uncased or all-minilm-l6-v2, the knowledge is solid. For obscure forks, it's less reliable — which is why the quality contract v2 fields in the article pipeline don't carry over to model entries the same way. I accept that model entries are "Claude's best knowledge" rather than verified first-person observation, and I label them model_used = claude-haiku-4-5.
Parsing and the upsert
JSON extraction follows the same regex-with-fallback pattern I use across all three ETLs:
function parseOrFallback(text: string, fb: GeneratedContent): GeneratedContent {
try {
const jsonMatch = text.match(/\{[\s\S]*\}/);
if (!jsonMatch) return fb;
const parsed = JSON.parse(jsonMatch[0]);
return {
summary: parsed.summary ?? fb.summary,
use_cases: Array.isArray(parsed.use_cases) ? parsed.use_cases : fb.use_cases,
pros: Array.isArray(parsed.pros) ? parsed.pros : fb.pros,
cons: Array.isArray(parsed.cons) ? parsed.cons : fb.cons,
};
} catch {
return fb;
}
}
The ?? and array checks make this tolerant of partial JSON — Claude sometimes drops a field when it genuinely doesn't know the answer, and that's fine as long as the rest is valid.
The write is an upsert:
INSERT INTO model_content (model_id, summary, use_cases, pros, cons, generated_at, model_used)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(model_id) DO UPDATE SET
summary = excluded.summary,
use_cases = excluded.use_cases,
pros = excluded.pros,
cons = excluded.cons,
generated_at = excluded.generated_at,
model_used = excluded.model_used
The ON CONFLICT DO UPDATE means the pass is fully idempotent — it can re-run tonight and tomorrow without duplicating rows. This is the same upsert shape I hit issues with in the OSS Alternatives ETL, though that article was about a different conflict condition.
After the upsert, the ETL exports the updated records to models.json for Astro's static build. The compare pages that generate pairwise model comparisons read from the same JSON file, so upgraded entries automatically propagate to those pages on the next build.
What actually changed in the upgrade run
The nightly pass that ran on August 16 upgraded 43 entries. A few examples of what changed:
MiniMax-H3 (image-text-to-video): From "MiniMax-H3 is an open-source image-text-to-video model available on HuggingFace. Details are sourced from the public model registry" to: "MiniMax-H3 generates synchronized audio-video clips from text or image prompts, producing output with coherent ambient sound and motion together. It supports multiple input modalities including text-to-video, image-to-video, and video-to-video transformation pipelines. The model ships on Diffusers and uses safetensors checkpoints, making it straightforward to integrate into ComfyUI or custom generation workflows." The use cases went from generic ("Building image-text-to-video applications") to specific ("Generating short video clips with synchronized ambient audio from text prompts", "Video-to-video style transfer with audio preservation").
ResNet-50 (RAM variant): From "is an open-source image-classification model" to a summary that names the Repeated Augmentation Masking training recipe and references the arxiv paper. The cons section now correctly notes the fixed 224×224 input resolution and the lack of attention mechanisms — things you'd actually want to know before choosing between this and a ViT model.
The overall pattern: Claude identifies the architectural details, training recipe, license constraints, and practical gotchas that a developer would want before deciding whether to try the model. The fallback content couldn't do this because it had no per-model knowledge.
Check any of the upgraded entries at /models/sentence-transformers-all-minilm-l6-v2/ on aiappdex.com — the content format is the same across all entries. The seeded variant generation writeup covers how the initial JSON seeds got into the database in the first place.
What I'd do differently
Sequential API calls are the bottleneck. The current loop calls Claude once per model, waits for the response, then writes, then calls again. For 43 entries, that's fine. For 400, it starts to matter. The Anthropic Batch API would let me send all 43 in one request and get results back asynchronously. I haven't switched because the upgrade pass only runs when there are fallback entries to fix — it's not a steady-state workload. But if the directory grows to 50,000 models, batch mode becomes necessary.
System prompt caching. The system prompt is 130 tokens. With prompt caching, that would be a cache hit on every subsequent call in the same batch. Right now the loop doesn't pass cache_control on the system prompt because it's short enough that the savings are minor at this scale. At 500+ entries per run, I'd add it.
Model card fetching. For top-N entries by downloads, it would be worth fetching the actual HuggingFace model card and including it in the prompt. The current approach relies on Claude's training knowledge, which is good for established models but weak for recent or obscure ones. The tradeoff is latency and rate limits from HuggingFace. I'll add this for entries above 10M downloads where accuracy matters more.
The pass currently costs roughly $0.04 per 43 entries at Haiku 4.5 pricing — negligible, but worth tracking as the directory grows.
FAQ
How do you know template content persisted and wasn't just a bad Claude response?
The model_used field is the signal. When Claude returns valid JSON, the field is set to claude-haiku-4-5. When the fallback activates (no API key, Claude error, parse failure), the field is fallback-template. The detection SQL only targets these fallback-marked entries, not entries where Claude ran and produced short or imprecise content.
Can the upgrade run overwrite good Claude-generated content?
No. The SQL detection query only selects WHERE model_used IN ('fallback-template', 'seeded-from-json'). Entries where Claude already ran and produced model_used = 'claude-haiku-4-5' are excluded. The only way an already-upgraded entry gets re-run is if I manually reset its model_used to fallback-template.
Why export to JSON instead of querying the database at build time?
Astro's static build runs getStaticPaths() at build time and needs deterministic data. If I queried the Turso database at build time, I'd need the DB credentials available to the Vercel build environment, and a DB error would break the build. The ETL-to-JSON pattern keeps the build offline-capable and reproducible. The downside is a one-build lag before changes appear in production — acceptable for a directory that refreshes nightly.
What's the "seeded-from-json" model_used value?
Early in the project, before the full Claude content generation pipeline was running, I seeded the database from static JSON snapshots of the HuggingFace API. Those entries got basic metadata (name, downloads, pipeline tag) but no generated content. The seeded-from-json tag marks them so the upgrade pass can find them alongside the fallback-template entries.
Does this create duplicate entries in the compare table?
No. The compare ETL (which generates pairwise comparison pages) uses a pair_slug as the unique key. Upgrading a model's content doesn't change its slug or ID, so the compare entries aren't affected. The next compare ETL run will regenerate comparison text using the upgraded content where it reads model summaries.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
