ClauseWatch: an agent that refuses to give you one number

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

Ask any model how long you must keep AI system logs under the EU AI Act. It will say six
months. That is true, and it is not the answer.

• AI Act Art. 19 makes the provider keep logs at least six months.
• AI Act Art. 26(6) puts the same floor on the deployer — a separate duty on a separate party.
• GDPR Art. 5(1)(e) says personal data may be kept no longer than necessary, and names no number at all.
• AI Act Art. 12(3) guarantees the collision: for biometric systems the log is required to record the identity of the humans who verified a match. The log the AI Act mandates is personal data by construction.

One instrument sets a floor. Another sets a ceiling. Neither says where the other one sits.
The honest answer is a period you can justify against both — plus a record of who decided it.

ClauseWatch gives that answer. It reads a structured model of obligations and the prose of the
instruments behind them, and when two sources pull in opposite directions it shows both with
their citations and reports that the decision is unmade:

Both clauses stand until someone signs. The dataset keeps a decidedBy and a decidedAt
field for exactly that, and they are empty.

This is the part that needed structured content. A keyword search over the same texts returns
Art. 19 and stops. It cannot know that a clause in a different regulation constrains the
same artifact from the opposite direction, because that fact is not inside either document.
It is in the relationship between them — which is to say, in the schema.

The control: I ran the keyword search

The brief sets a bar — if a keyword search would have gotten you the same answer, aim
higher — so rather than assert we cleared it, here is the other arm of the experiment.
Ordinary tf-idf over the same clause texts the knowledge base is built from:

$ npm run ask -- --baseline "how long must I keep automatically generated logs"

  0.2763  EU AI Act Art. 19     — Automatically generated logs
  0.2763  EU AI Act Art. 26(6)  — Deployer duty to keep automatically generated logs

Two clauses, identical scores. Ranking cannot break that tie, because which one binds
you is not a property of either text — it depends on whether you are the provider or the
deployer, which is a fact about you.

And GDPR Art. 5(1)(e) is not in the results at all. It is the clause that makes the answer
wrong if you ignore it, and it is invisible to this query because it never says logs. It
says personal data:

$ npm run ask -- --baseline "how long may I keep personal data"

  0.2865  GDPR Art. 5(1)(e)     — Storage limitation
  0.1066  EU AI Act Art. 26(6)  — Deployer duty to keep automatically generated logs
  0.0759  EU AI Act Art. 19     — Automatically generated logs

The two halves of the answer live under different vocabularies. Finding both requires
already knowing both exist, which is the question you were asking in the first place.

Asking about the collision directly returns one clause, because the collision is not in any
document. It is a relationship between two of them, and a flat index of documents has
nowhere to put a relationship. That is the whole argument for modelling it, and it is now a
command anyone can run rather than a claim in a blog post: the full comparison is in
demo/05-keyword-baseline.md.

Demo

git clone https://github.com/oleg-vdv/clausewatch && cd clausewatch/web
npm install && npm run dev # http://localhost:4173

No credentials. No login, no token, no API key. The dataset is public, so a fresh clone
reads it over the public query API and renders the full report. The knowledge base and the
LLM agent do need a Context token, which you cannot have — so --check and the page footer
say which of the two sources the answer came from, rather than crediting one they never
touched.

It also runs with no model at all:

cd ../agent && npm install
npm run ask -- --profile biometric-access --no-llm   # provider, high-risk, KZ → EU
npm run ask -- --profile support-agent --no-llm      # deployer, risk class unset

Two elements of the viewer do the arguing.

The pressure bar — a solid edge on the left where Art. 19 states six months, and a
hatched, edgeless right where the GDPR caps the period without naming one. Drawing a tidy
range there would be a lie about the law.

The signature block — every conflict prints a decided by and a date rule. Resolved,
they carry a name and a date. Open, they are two empty lines in a compliance report. The
thesis as a form field rather than a paragraph.

The Studio is live at https://clausewatch.sanity.studio/, with open conflicts on the first
screen, because that is the one thing an editor of this dataset actually does.

Code

https://github.com/oleg-vdv/clausewatch

studio/   schema, desk structure, seed dataset
agent/    MCP client, domain layer, LLM agent, deterministic report
web/      the viewer — Astro, server-rendered, sharing the agent's domain layer
demo/     saved runs with their tool calls attached

The content model

source        an instrument: binding status, official URL, version label
provision     one citable unit: "Art. 19", verbatim text, effective date, roles bound
requirement   a normalised obligation: "retain automatically generated logs"
  └ claim[]   what each instrument asserts — direction (floor/ceiling/none), period if
              stated, condition, and a reference to its provision
conflict      two clauses that cannot both be satisfied, plus resolution, rationale,
              decidedBy, decidedAt
systemProfile roles, jurisdictions, risk class, agentic or not

Three decisions carry the weight.

Claims sit apart from requirements. One obligation, many instruments, and disagreement
between them is content — not a data-quality problem to be cleaned up.

direction on a claim. An earlier version bucketed claims by does it state a number,
which filed GDPR Art. 5(1)(e) under silence. It is not silent: it caps the period without
naming one. Floor, ceiling and duty-only are three different answers.

A conflict holds a decision, not a resolution rule. An agent that silently picks the
stricter number is guessing on your behalf. One that shows both sides and cites a named,
dated decision produces something an auditor can accept.

How I Used Sanity

What I pointed Sanity Context at

Two website sources, both crawled to exact paths rather than wildcards:

26 documents, indexed into 10 entries. Curated on purpose — and also capped, because the
Context beta allows 150 indexed documents on this plan and I hit that twice before narrowing
properly.

The indexing surprised me. Entries are topic nodes rather than chunks: each carries an article
range, a topic list, cross-references (excludes: … see <other entry>) and a numbered Sources
block with a URL behind every statement.

Which Context tools I used

Two endpoints, because one endpoint serves one mode. Attaching a Knowledge Base to an
endpoint that already served a dataset replaced the GROQ tools entirely.

Both carry the same endpoint Instructions — the citation discipline the agent must follow,
delivered before it sees any data. The rule that matters most:

Never fill a gap from your own legal knowledge. If it is not in the dataset or the knowledge
base, say that it is not there.

Beyond the tools I leaned on two Context features: Issues, which surfaced a contradiction
in the corpus, and Instructions, which corrected a fact at the source instead of in my
code. Both stories are below.

What the agent actually did with it

A real tool-use loop. Nothing is pre-fetched: the model gets the two endpoints and has to go
and look. A typical run:

1. initial_context         (dataset — schema and the citation rules)
2. kb_initial_context      (knowledge base — the outline of 10 entries)
3. groq_query              (the logging requirement, its claims, its conflicts)
4. kb_knowledge_base_read  (design_documentation, post_market_obligations)
5. groq_query              (jurisdictions, extraterritoriality)
6. kb_knowledge_base_read  (high_risk_categories)
7. groq_query              (system profiles)

Seven calls across both endpoints. The answer states it cannot give one number, shows the
floor and the ceiling with citations and effective dates, reports the conflict as open with
nobody named, and notes the obligation does not bite until 2 December 2027. In a second run
the agent found the matching system profile unprompted, treated riskClass: unknown as
load-bearing, separated provider duties from deployer ones, and closed with a section titled
What I can't answer from these sources.

Saved runs with their tool calls are in
demo/. The answer alone cannot
show that a number came from the endpoints rather than from the model, so the tool calls
travel with it.

The dataset side does the reasoning that a prose search cannot: claims are filtered by the
jurisdictions a system touches and the roles it holds, so a deployer is shown Art. 26(6)
and a provider Art. 19. Telling a deployer that Art. 19 is their duty is not a rounding error,
it is the wrong party.

Context found a contradiction I had not

While indexing, Context flagged that an entry claimed Annex I lists 21 harmonisation
instruments while the source shows 20 — item 1 deleted by amendment, item 21 added. I checked
against the source: true. It also proposed a Section A/B split that I could not verify,
because EUR-Lex blocks automated retrieval.

So the decision recorded in the dataset says the detection was right, states only the verified
part, explicitly excludes the unverified arithmetic, and carries a name and a date.

That is the whole product in one incident. Automated detection is good at finding
disagreement and is not authoritative about resolving it.

Where prose alone gets a citation wrong

In one run the agent cited Art. 26(5) for the deployer's log-retention duty, from the
knowledge base. The source says Art. 26(6); 26(5) is the monitoring duty. A one-digit
error that reads as correct.

Chasing it was the most useful hour of the build. The knowledge-base entry for Article 26
numbers its own sections, and the fifth one is headed ### 5. Log retention. Section five of
an entry about Article 26 — and a model reading it produced "Art. 26(5)". The entry states the
rule itself correctly, and gives no paragraph number at all:

Deployers shall keep automatically generated logs for a period appropriate to the intended
purpose, with a minimum of six months, unless applicable Union or national law … provides
otherwise.

That is the argument for two layers rather than one. Prose is right about the rule and
silent on the address; the dataset carries the address, verified by a human against the
source. Art. 26(6) is now a provision with the checked citation, its own claim on the
retention requirement, and the role it binds — which is why a deployer profile is shown
Art. 26(6) and a provider Art. 19.

I also added a Context Instruction, which is honored on every build, over the raw
sources, so a future rebuild cannot reintroduce the wrong number. Being precise about what
I have and have not verified: that Instruction has not been exercised yet. The corpus has not
changed since, so Context reports entries up to date and no rebuild has run.

The layers check each other in both directions. In another run the agent found a duty present
in the prose that my dataset had not modelled at all, and said so.

Here is the check itself, thirty seconds of it: a script pulls Article 26 from the source and
prints its numbered paragraphs. Paragraph 5 is the duty to monitor. Paragraph 6 is the duty to
keep the logs. That is the whole difference between citing the right clause and the wrong one,
and nothing short of going to the text would have shown it.

Embedded session cards open collapsed — click the ▸ to see the command and everything it
printed. Same for the three further down.

$ node "/c/Users/Admin/AppData/Local/Temp/claude/C--Users-Admin-Documents/ddd0a186-49e0-40f7-be9b-b7de1c0d2f4d/scratchpad/art26.mjs" 2>&1 | head -40

[para 1] . Deployers of high-risk AI systems shall take appropriate technical and organisational measures to ensure they use such systems in accordance with the instructions for use instructions for use means the information provided by the provider provider means a

[para 2] . Deployers shall assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support. 3. The obligations set out in paragraphs 1 and 2, are without prejudice to other deployer deployer means

[para 4] . Without prejudice to paragraphs 1 and 2, to the extent the deployer deployer means a natural or legal person, public authority, agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-p

[para 5] . Deployers shall monitor the operation of the high-risk AI system AI system means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objec

[para 6] . Deployers of high-risk AI systems shall keep the logs automatically generated by that high-risk AI system AI system means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment,

[para 8] . Deployers of high-risk AI systems that are public authorities, or Union institutions, bodies, offices or agencies shall comply with the registration obligations referred to in Article 49 . When such deployers find that the high-risk AI system AI system mea

Five things that cost me hours

• Document ids containing dots are invisible to anonymous readers, even in a public dataset. The CLI said 18 imported, an authenticated count(*) said 31, an anonymous one said 0. Sanity treats _id as a path and public read covers the root path only. An authenticated count is not evidence that your dataset is public.
• Wildcard include patterns do not filter a website source; exact paths do. /article/* pulled 200 pages including Polish and French translations of the same articles. For a legal agent a translation is a correctness hazard, not noise — and the sitemap is the tell: if a crawl returns more pages than the sitemap lists, it followed in-page language links.
• "Sitemap only" overrides include patterns rather than narrowing them. 691 documents.
• An entry outlives its sources. Delete a source and the prose stays while every citation becomes _source no longer available — and the entry is still listed in initial_context for the agent to read and cite. Dismissing the issue keeps the entry; only a rebuild cleared them.
• Context rewrites your GROQ, which changes result shapes: roles[]->name comes back as [{name, _id}]. It also injects _type != "sanity.agentContext" and pins perspective: published. Worth knowing when a query behaves differently in Vision.

And one that is nobody's fault but mine: count() of a missing field is null, not 0, so a
role filter written as count(appliesToRoles) == 0 for binds everyone silently dropped every
GDPR claim. It looked fine in the provider's report and only broke for the deployer.

Watching the first one happen

The numbers are the whole story, so here is the terminal output: an import that reports
success, a document fetch that comes back {"documents":[],"omitted":[{"reason":"permission"}]},
a one-document probe with a plain id that proves the ids are the cause, and a final anonymous
count(*) of 18 once everything was renamed to hyphens.

$ cd /c/Users/Admin/projects/clausewatch/studio && set -a; . ../.env; set +a
echo '{"_id":"anontest-nodots","_type":"jurisdiction","name":"Anon probe","code":"ZZ"}' > /tmp/probe.ndjson
npx sanity dataset import /tmp/probe.ndjson production --replace 2>&1 | tail -2
sleep 2
echo "=== anonymous query ==="
curl -s "https://4yzoidsq.api.sanity.io/v2026-09-19/data/query/production?query=$(node -e 'process.stdout.write(encodeURIComponent(`*[_type=="jurisdiction"]{_id,name}`))')"; echo

Done! Imported 1 documents to dataset "production"

=== anonymous query ===
{"query":"*[_type==\"jurisdiction\"]{_id,name}","result":[{"_id":"anontest-nodots","name":"Anon probe"}],"syncTags":["s1:pQbSuQ"],"ms":1}
Shell cwd was reset to C:\Users\Admin\Documents

Sanity Project Details

• Project ID: 4yzoidsq
• Dataset: production (public)
• Studio: https://clausewatch.sanity.studio/

Public dataset, no token — every conflict, both sides, with its decision:

https://4yzoidsq.api.sanity.io/v2026-09-19/data/query/production?query=*[_type=="conflict"]{summary,nature,resolution,decidedBy,"sides":sides[]->{"cite":source->shortName+" "+citation}}

Agent Session

The whole build is on record — 485 messages in one Claude Code session. A note before you
open it: I work in Russian, so the conversation is in Russian. Everything that matters here
is not. The tool calls, the GROQ queries, the MCP traces, the terminal output and the agent's
own answers are all English, and that is what each slice below is made of. I have said what
to look for in front of each one — and each card opens collapsed, so click the ▸ to see the
output rather than just the command line.

The agent refuses to give one number

The question is how long a provider must keep logs for a high-risk biometric system. Watch
the trace: initial_context on the dataset, then on the knowledge base, then a GROQ query it
writes itself, then two knowledge-base entries, then jurisdictions, then profiles. Seven
calls across both Context endpoints before it says anything.

Then read the answer. It gives a floor with a citation, a ceiling with a citation, states
that the conflict between them is unresolved and that nobody is recorded as having decided
it, and points out that the obligation does not apply until December 2027. It never produces
the single number the question was fishing for.

$ cd /c/Users/Admin/projects/clausewatch/agent && npm run ask -- "How long must the provider of a high-risk biometric access-control system keep its automatically generated logs, and from when?" 2>&1 | tail -50

> clausewatch-agent@0.1.0 ask
> tsx src/cli.ts How long must the provider of a high-risk biometric access-control system keep its automatically generated logs, and from when?

  → data · initial_context
  → docs · initial_context
  → data · groq_query
  → docs · knowledge_base_read
  → data · groq_query

## Log retention for a high-risk biometric access-control system's provider

**The floor: at least six months (EU AI Act, Art. 19)**

Under EU AI Act **Article 19**, providers of high-risk AI systems must keep the automatically generated logs (as produced under Article 12(1)) "for a period appropriate to the intended purpose of the high-risk AI system, of **at least six months**, unless provided otherwise in the applicable Union or national law, in particular Union law on the protection of personal data" — and only to the extent the logs are under the provider's control.

- Source: EU AI Act, Art. 19 — official text: https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- **Effective date:** this obligation (like Art. 12 logging design duties) enters into force **2 December 2027** for Annex III high-risk systems, or **2 August 2028** for Annex I high-risk systems. As of today (2026‑09‑19) it is **not yet a current obligation** — it is a future one.

For a biometric identification system specifically (Annex III, point 1(a)), Article 12(3) additionally requires the logs to capture, at minimum: usage period (start/end date-time of each use), the reference database checked, the matched input data, and the identity of the human verifier(s) involved — but Art. 12(3) itself does not add a separate retention period; the six-month floor still comes from Art. 19.

**The ceiling that conflicts with it: GDPR Art. 5(1)(e)**

The dataset records an unresolved conflict on this exact point:

> "AI Act sets a six-month floor for log retention; GDPR sets a necessity ceiling on the same logs"

- **AI Act side:** Art. 19 — "kept for a period appropriate to the intended purpose … of at least six months" (as above).
- **GDPR side:** Art. 5(1)(e), Storage limitation — "Personal data shall be kept in a form which permits identification of data subjects for **no longer than is necessary** for the purposes for which the personal data are processed." (GDPR official text: https://eur-lex.europa.eu/eli/reg/2016/679/oj)

The dataset notes that because biometric logs under Art. 12(3) necessarily contain personal data (matched biometric input, verifier identity), this ceiling is essentially unavoidable for this system type.

**Resolution status: open.** The dataset marks this conflict's `resolution` as **"open"** — it has no `rationale` and no `decidedBy` recorded, meaning it has not been settled. I can't tell you who is designated to decide it, because the dataset only records a decider for resolved conflicts, and this one isn't resolved.

**Bottom line:** you cannot state a single fixed retention number. The provider must retain the logs for **no less than six months** (Art. 19 floor) **and no longer than necessary for the purpose** for which the biometric data is processed (GDPR Art. 5(1)(e) ceiling), with no dataset-recorded resolution telling you how to reconcile the two if "necessary" turns out to be shorter than six months. Both duties only become live obligations from **2 December 2027 / 2 August 2028** depending on whether the system falls under Annex III or Annex I — not before.

One more caveat per the rules I follow: this assumes your system's profile (role = provider, jurisdiction = EU, risk class = high, EU-market use) is confirmed. I haven't been given a system profile in this conversation — if any of those facts differ, the applicable obligations could change.

(5 tool calls)
Shell cwd was reset to C:\Users\Admin\Documents

The Annex I check, start to finish

This one starts with a Sanity Context issue: an entry says Annex I lists 21 harmonisation
instruments, the source shows 20. You will see me go to the source rather than take it on
trust — the curl that pulls the annex straight from the source, and the amendment markers
in its output: item 1 struck, item 21 added.

The end of the slice is the part I would point a judge at. Context also proposed a Section
A / Section B split, and I could not verify it, because EUR-Lex blocks automated retrieval.
So the decision recorded in the dataset states the verified part and explicitly excludes the
rest. Finding disagreement and settling it are different jobs.

$ curl -s https://artificialintelligenceact.eu/annex/1/ | node -e '
let d="";process.stdin.on("data",c=>d+=c).on("end",()=>{
  const txt=d.replace(/<script[\s\S]*?<\/script>/g,"").replace(/<style[\s\S]*?<\/style>/g,"")
             .replace(/<[^>]+>/g," ").replace(/&#8217;|&rsquo;/g,"’").replace(/&nbsp;/g," ")
             .replace(/\s+/g," ");
  const a=txt.indexOf("Section A");
  const b=txt.indexOf("Section B");
  console.log("--- around Section A ---");
  console.log(txt.slice(a, a+300));
  console.log("\n--- tail of Section A / start of Section B ---");
  console.log(txt.slice(b-500, b+300));
  console.log("\n--- last 900 chars of annex body ---");
  const end=txt.indexOf("Feedback")>0?txt.indexOf("Feedback"):txt.length;
  console.log(txt.slice(Math.max(0,end-1200), end));
});'

--- around Section A ---
Section A. List of Union harmonisation legislation based on the New Legislative Framework Related recitals: 50 51 1. Not present after the amendment. Directive 2006/42/EC of the European Parliament and of the Council of 17 May 2006 on machinery, and amending Directive 95/16/EC ( OJ L 157, 9.6.2006, 

--- tail of Section A / start of Section B ---
I Arts. 57–63 Measures in Support of Innovation Article 57: AI Regulatory Sandboxes Article 58: Detailed Arrangements for, and Functioning of, AI Regulatory Sandboxes Article 59: Further Processing of Personal Data for Developing Certain AI Systems in the Public Interest in the AI Regulatory Sandbox Article 60: Testing of High-Risk AI Systems in Real World Conditions Outside AI Regulatory Sandboxes Article 60a: Testing of high-risk AI systems covered by Union harmonisation legislation listed in Section B of Annex I in real-world conditions outside AI regulatory sandboxes Article 61: Informed Consent to Participate in Testing in Real World Conditions Outside AI Regulatory Sandboxes Article 62: Measures for Providers and Deployers, in Particular SMEs, Including Start-Ups Article 63: Derogati

--- last 900 chars of annex body ---
, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments Article 3(1) or a general-purpose AI model on the Union market Article 3(9) of aircrafts referred to in Article 2 (1), points (a) and (b) thereof, where it concerns unmanned aircraft and their engines, propellers, parts and equipment to control them remotely, are concerned. 21. Regulation (EU) 2023/1230 of the European Parliament and of the Council of 14 June 2023 on machinery and repealing Directive 2006/42/EC of the European Parliament and of the Council and Council Directive 73/361/EEC ( OJ L 165, 29.6.2023, p. 1 , ELI: http://data.europa.eu/eli/reg/2023/1230/oj ). Not present before the amendment. new View in the full AI Act Explorer Annex II Cookies on this website We use cookies to understand how the explorer is used. Functional cookies are always on; everything else is your choice, shared with artificialintelligenceact.eu so you are only asked once. Read the Cookie Policy and the Privacy Statement . Functional (always on) Statistics Marketing Preferences Accept all Reject all Save preferences Manage preferences

Honest limits

The knowledge base is built from authoritative reproductions, not the Official Journal —
EUR-Lex blocks automated retrieval. Every source carries an officialUrl and the agent is
instructed to give that one for anything the reader will act on.

The dataset is small and deliberately so: 2 instruments, 9 provisions, 2 requirements,
3 conflicts, 2 system profiles. A working core that demonstrates the model, not a compliance
product. Nothing here is legal advice.