by shahbaz_ali. Built for the DEV × Sanity Challenge. #sanitychallenge
When two of your company's documents disagree, somebody has to decide which one wins. Usually that's a Slack thread, and usually it's forgotten by the time the same question comes around again.
I built a tool that turns that decision into a typed, queryable document with a source, a reason, a precedent chain and an audit trail. An agent proposes a ruling. A human approves it or overrides it. Then the answer changes everywhere at once.
Everything a judge needs is right here, so I'll put it up top.
- Live app: https://www.sanity.io/@ouwae45xd/application/szyxap501o5s89a8yhoq5l8c
-
Judge login:
sanity.judges@tokmail.net/Shahbaz123. -
Sanity project ID:
cqb58l0f - Public dataset (GROQ queryable): https://cqb58l0f.api.sanity.io/v2025-01-01/data/query/production?query=*[_type=="instruction"]{_id,resolution,"topic":appliesToTopic->name}
The App SDK runs inside the authenticated Sanity Dashboard, which means the app wants a login. The account above is a dedicated Editor-role one I set up for judges, so nobody has to wait on an invite.
The Triage tab showing three conflicts across three topics
If the recording isn't your thing, the steps are written out further down and take about two minutes.
Why I wanted this to exist
Company knowledge lives in documents written by different teams. Legal writes the terms of service, support writes help articles, marketing writes the pricing page, and policy writes the internal rules. Sooner or later two of them disagree.
Say the internal policy promises refunds within 30 days. The public FAQ says 14. Marketing, for reasons nobody remembers, says 45. Which one is right?
At most companies the answer comes out of a Slack thread. Someone asks someone else, somebody says "I think 30 is right," and everyone moves on. Three months later a new hire asks the same thing. The thread is buried under 10,000 other messages, the FAQ still says 14 days, and nobody can tell you why the call was made or who made it.
Contradiction Triage is my attempt at fixing that. Every decision becomes a first-class document with provenance, reasoning, and the ability to be cited by later decisions. Anyone who asks gets the current answer.
The loop
A contradiction gets detected and a "case" document is created. An agent proposes a ruling, with a rationale and cited precedents. A human approves it or overrides it. Once the human rules, that decision is the canonical answer the app serves.
There are three tabs. Triage is the work queue, grouped by stage: "Awaiting your ruling," "Needs a proposal," "Recent rulings" and "Superseded." History lists every ruling ever made, with attribution. Answers shows the canonical answer for each topic and where it came from.
The schema is the feature
There are six document types, and I spent more time on them than on any UI code.
A source is a document of record: title, sourceType, url, content, lastReviewedAt. A topic is the subject a claim is about. A claim is a statement with a normalized value like "30 days," a source reference, a topic reference and a confidence score. A case is a detected contradiction, holding a topic, a snapshot of the conflicting claims and an optional agent proposal. A caseEvent is an append-only log entry for every stage transition, with the actor who made it (human, agent, system or seed) and a payload snapshot.
And then there's instruction, which is the ruling itself. It has a winner (a claim reference) or an outcomeValue (a string, for the times a ruling lands on a value no claim asserted). It also carries overruled[], typed precedents[], and supersedes, a weak reference to whichever instruction it replaces.
Four decisions in there do most of the work.
topic is a document, not a string
Both claim.topic and instruction.appliesToTopic point at the same topic document. So a ruling can't quietly fail to apply because two strings drifted apart.
Without that, "refund-window", "refund window" and "Refund-Window" would be three different topics. A ruling on one wouldn't answer the other, and nothing would warn you. The app would just show topics that look related and aren't.
Values are comparison keys
Each claim carries a normalized value, like "30 days", "5 USD" or "12 months". Two claims on the same topic with different values contradict each other. Two claims with the same value agree, and the app leaves them alone.
It's the difference between counting and comparing. If all I had was "how many unresolved claims sit on this topic," two sources saying the same thing would look exactly like a conflict.
Precedents are typed
When a ruling cites an earlier ruling, the citation records how it was used.
-
followsmeans the earlier ruling applies here too. -
distinguishesmeans the earlier ruling looks applicable but isn't, and here's why. -
overrulesmeans the earlier ruling was wrong and we're reversing it.
A flat reference array can't say any of that, and the difference isn't academic. The shipped dataset has a case built specifically to show distinguishes. The agent cited an earlier ruling as follows. The recorded human ruling cited the same one as distinguishes. Both landed on the same answer. The conclusion survived and the argument didn't, and you can only see that because the precedents are typed.
Supersession is recorded, not inferred
When a new ruling replaces an old one, the new instruction carries a supersedes reference. The old ruling never gets deleted. "Is this instruction superseded?" is answered by count(*[_type == "instruction" && supersedes._ref == ^._id]) > 0.
The gap between that and a naive references() check cost me a real bug, which I'll get to below.
The state machine is data
workflow.def.json sits at the repo root and declares four stages (detected → proposed → ruled → superseded) plus the transitions between them, including which kinds of actor may fire each edge. The seed script validates every event it writes against that table. The UI reads the same table to decide which controls to enable.
One invariant is enforced structurally. ruled can only be fired by a human. No agent, script or system path can commit a decision.
Why the App SDK and not a Studio plugin
The challenge offers bonus points for reaching past the Studio, and I'd be lying if I said that had no pull. But the real reason was the write path. What's interesting here isn't the display, it's what happens when someone clicks Approve.
A Studio plugin renders a form. As far as I can tell, an App SDK app can drive a whole state transition atomically, with optimistic updates, through the same SDK the Studio itself runs on.
Every ruling is one transaction. Approve, override-with-a-different-claim and override-with-a-new-value all funnel through a single writeRuling() function, which emits the instruction, the event and the legacy claim sync as one batch. If any part fails, nothing lands. The state can't end up half-updated.
await apply([
createDocument(instructionHandle, instruction),
publishDocument(instructionHandle),
editDocument(winnerHandle, {set: {status: 'resolved'}}),
...overruledHandles.map(h => editDocument(h, {...})),
...overruledHandles.map(h => publishDocument(h)),
createDocument(eventHandle, caseEvent),
publishDocument(eventHandle),
])
Two proposers, one output shape
The agent comes in two flavors, and both write the same case.proposal fields: outcome, rationale, confidence, precedents, proposedAt, model and promptVersion. The only tell is the model field.
The first is deterministic, in scripts/lib/propose.mjs. It's a pure function that ranks claims by source authority (official beats internal beats external beats community), then by the source's review date, then by claim confidence, then lexicographically by claim id. It runs in the browser behind the "Propose ruling" button and needs no API key. Its confidence comes from whichever rule decided the case: 0.9 for authority, 0.75 for recency, 0.6 when claim confidence decides, and 0.5 for a coin-flip tiebreak.
The second is scripts/agent.mjs, which calls Groq's free tier with openai/gpt-oss-120b. It reads the case's conflicting claims plus prior rulings on the same topic, and writes a proposal with a rationale and cited precedents. It runs from the CLI because that's where the API key lives.
The rest of the pipeline
scripts/extract.mjs reads a source's prose, calls an LLM, and produces structured claims shaped like {statement, value, topicSlug, confidence}. It does a dry run by default and only writes with --commit.
scripts/detect.mjs scans unresolved claims, groups them by topic, and opens a case for any topic with at least two claims and at least two distinct values. No LLM involved, just a group-by.
scripts/seed.mjs writes 90 documents: 5 sources, 11 topics, 24 claims, 12 cases, 7 instructions and 31 caseEvents. It does that in five phases, and the reason is the most annoying thing I ran into on the Sanity side.
Content Lake validates references at write time, in both directions. You can't create a claim that points at an instruction that doesn't exist yet. You can't delete a document another document points at. And when two document types reference each other (claim to instruction, instruction back to claim), no creation order works in a single pass.
The fix is ugly but it works. Create everything without the cyclic fields, then patch them in. The fifth phase exists only to close the cycle the first four opened. I'm not sure it's the cleanest way to do it, but it's been stable, so I've stopped poking at it.
How I built it
The whole project went through Cline with DeepSeek V4 Flash. One phase at a time, and each phase had to pass npx tsc --noEmit --incremental false and npx sanity schemas validate before I moved on. That comes to roughly 40 prompts across 15 phases over a couple of days.
Here's the finding I'd hand to anyone starting something similar. sanity build does not type-check. It bundles and strips types. The first time I hit real type errors after a refactor, tsc exited 2 while sanity build cheerfully reported success. If you only run the build, you ship broken code with a green checkmark. Run both.
The three bugs
Three bugs hit during development and they all had the same root cause. Naming them as a class is probably the most useful thing I can write down here.
claim.resolvedBy was overloaded
An early version had one field on claims, resolvedBy, a reference to the instruction that settled it. It looked correct.
It wasn't. The field really encoded "overruled by," not "involved in a ruling." A winning claim carried no resolvedBy at all. But GROQ's references() matches winners and overruled claims alike, so any query deriving "is this claim resolved?" from references() would mislabel winners as losers.
The fix was to split the meaning in two. instruction.winner is singular and instruction.overruled[] is plural. After that, every query had to be role-aware, and every read of "which instruction settled this claim" had to say how.
references() was overloaded too
Later in the build, the canonical-answer query tried to find "the instruction nothing supersedes."
*[_type == "instruction"
&& appliesToTopic._ref == $topicId
&& count(*[_type == "instruction" && references(^._id)]) == 0
] | order(decidedAt desc)[0]
It returned 5 rows when it should have returned 6. One topic's canonical answer had silently vanished.
The problem is that references() matches any inbound reference. A later instruction that cited an earlier one as a precedent (precedents[].instruction) looked, to this query, identical to a later instruction that replaced it. The query had merged "cited by" with "replaced by."
The fix:
count(*[_type == "instruction" && supersedes._ref == ^._id]) == 0
Same class of mistake, same shape. One predicate, two meanings.
A comment swallowed an export
The third one lived in code instead of data. I was extending a docblock in src/queries.ts and dropped the closing */. The export const CANONICAL_ANSWER_QUERY line got eaten by the comment.
That is valid JavaScript. sanity build reported success, the app deployed, the app rendered. The only problem was that canonicalAnswerOptions() would have thrown a ReferenceError the moment anybody opened the Answers tab.
tsc caught it. Nothing else did.
What they have in common
Every one of these was something doing two jobs, quietly. A field meaning two things. A predicate matching two patterns. A comment swallowing code. The build caught none of them. tsc, a live GROQ query and a click-through caught all three.
In typed content models, I don't think the usual failure is "the type is wrong." It's that a mechanism is overloaded. Each time I split an overloaded thing, a whole class of bugs went away. Each time I tolerated an overload because it was convenient, a new bug showed up within a week.
The eval, and where the agent is wrong
The deterministic proposer is a total function, so it always makes a call. I scored it against the rulings on the 7 cases that have both a proposal and a ruling. (The dataset has 12 cases overall.) The report below is trimmed from what scripts/eval.mjs prints (my run was generated 2026-09-20T00:56:59Z):
OVERALL ACCURACY
Outcome match: 6 / 7 (86%)
Overrides: 1 / 7 (14%)
BY CONFIDENCE BUCKET
0.9-1.0: 5 / 5 (100%) - high-confidence
0.7-0.9: 1 / 1 (100%)
0.5-0.7: 0 / 0 (n/a)
0.0-0.5: 0 / 1 (0%) - coin flips
BY CATEGORY
authority-beats-recency: 3 / 3 (100%) - 0 overridden
newer-supersedes-older: 1 / 1 (100%) - 0 overridden
precedent-applies: 1 / 1 (100%) - 0 overridden
precedent-misleads: 1 / 1 (100%) - 0 overridden
true-tie: 0 / 1 (0%) - 1 overridden
CROSS-CHECK
Derived override agrees with recorded payload.approvedProposal: 7 / 7 (100%)
The headline number is the least interesting part. I care more about the shape.
The confidence buckets behave the way you'd hope. Every case at 0.7 or above matched the human, and the only miss sat in the lowest bucket, where the proposer is effectively saying "I'm flipping a coin." The confidence values come straight from which rule fired.
The precedent-misleads case is my favorite row. On case-gift-card-expiry-1 the agent cited the Account Deletion ruling as follows. The recorded human ruling cited it as distinguishes. Both landed on 12 months, so the outcome matched and it scores as a hit. But the argument was different, and the category is about the citation, not the outcome. An untyped reference array would have shown two rulings pointing at the same earlier ruling and called it agreement. That's the case that convinced me the schema was worth building.
The true-tie miss is a designed one. Two claims share source, review date and confidence, so nothing separates them. The proposer reports 0.5 and calls it a coin flip. The human ruling on Cancellation Notice established 45 days, a value neither claim asserted. So the agent proposed a claim and the human proposed a value, and the report itself flags this as a miss by construction: case.proposal has no field for a value. Counting that as a miss is honest, but the better reading is that the agent was right inside its scope, and this is exactly where its scope ends.
Then the cross-check, which is the line that lets me trust everything else. The override column is derived by comparing outcomes. The event log records the same fact independently, as payload.approvedProposal. They agree on 7 of 7. If they'd diverged, I'd have thrown out the numbers above.
Now the caveat, and it's a big one. This is 7 cases, and the rulings are seed fixtures I wrote myself, which the app labels "Seed fixture (simulated human)." So the ground truth is me playing reviewer, with one labeler. Five of the 12 cases couldn't be scored at all because they lack a proposal or a ruling. Read the categories as directions, not rates. The harness isn't there to prove the agent is right. It's there so the ways it's wrong are easy to see.
What isn't built
I'd rather say this plainly than have a judge find it.
Extraction and detection run from the CLI, not the UI. They need an API key and an outbound call to Groq, and a browser can't hold either (there's no safe place for the key in the bundle, and CORS blocks direct Groq calls anyway). Detection also writes to the dataset with a write token. The production wiring would be a Sanity Function triggered by a webhook when a source is created. The scripts read and write the same schema the UI does, so that's a deployment change and not an architectural one. To make the pipeline drivable from the app anyway, the UI ships with pre-computed extraction samples.
Value canonicalization is missing. 12 months and 1 year are the same assertion, and today the detector treats them as different. The right fix is probably a normalized {amount, unit} shape, or a canonicalization step at write time. This is the extractor's weakest point and I know it.
Detection only sees unresolved claims. A fresh source that contradicts an already-ruled answer is invisible right now. That's arguably the most interesting product case, and closing it means comparing new claims against the current canonical answer. I'd call that a design decision, not a bug fix.
Multi-tenancy isn't there. Topics are global. A multi-company deployment would add a company reference to topic and push it through every query. The governance loop itself wouldn't change, it would just gain a scope filter.
Workflows isn't integrated. I spiked Sanity's Workflows prerelease before building anything else, and it hit a peer-dependency wall. @sanity/workflow-sdk@0.33.0 requires @sanity/sdk-react@^3.1.0, and my working app is on 2.20.2. Bumping the SDK by a major version 10 days before the deadline would have put a working app at risk for a bonus criterion, so I killed it. I designed the scripts and schema so that swapping the hand-rolled state machine for the Workflows engine should be a data migration and not a rewrite.
Try the whole pipeline
Open https://www.sanity.io/@ouwae45xd/application/szyxap501o5s89a8yhoq5l8c and sign in with sanity.judges@tokmail.net / Shahbaz123. It's a shared workspace, so whatever you click is visible to the next visitor.
- Click Triage, then Warranty Period. Read the agent's proposal, its confidence and its rationale.
- Click Approve. Watch the case move to "Recent rulings," History gain an entry, and the Answers tab update the canonical answer for Warranty Period.
- If you want the full pipeline, click + Add Source, load the "Refund Policy v2" sample, and click Create Source + Claims. The app writes the source, extracts its claims (pre-computed, since extraction needs a server-side key), detects a fresh contradiction, and opens
case-refund-window-2, a 3-claim conflict between 14, 30 and 45 days. - Click the new case, hit Propose ruling, and watch the deterministic proposer rank the claims right there in the browser.
Source: https://github.com/ShahbazVK/SanityChallenge
The public dataset is queryable right now, so every schema claim in this post can be verified against it with GROQ. The queries above are the ones the app itself runs.
What the whole thing is really about
I've spent a lot of words on schemas and bugs, so here's the part I care about most.
A decision is content. It isn't a comment on a document or a Slack message or a line in a changelog. It's a document with a source, a reason, a precedent chain and an audit trail. It can be cited by a later decision. It can be superseded without being deleted. It can be served, untouched, to any system that needs a canonical answer.
The agents and the workflow engine are interesting, and I had fun with both. But the thing this project actually shows is simpler. If you model the decision properly, the decision becomes queryable. Once it's queryable, it's usable, by a person or by another agent, and nobody has to go digging through a Slack archive.
That's the bet. The rest is implementation.
Built for the DEV × Sanity Challenge. #sanitychallenge. Source was vibe-coded with Cline and DeepSeek V4 Flash.
