AladdinAI + Sanity Context: an agent that actually understands its own architecture

My submission for Path One of the Sanity Challenge an AI agent that queries a real Sanity dataset via MCP to answer questions about its own gate/model/trace architecture.

The idea

AladdinAI is a self-hosted, bring-your-own-infrastructure AI agent platform. One of the less obvious parts of its architecture is a layer of small classifier models I call them "gates" that sit at every data-transfer point between agents: handoffs, memory writes, and memory recall. Each gate uses a specific model, and every agent run produces a trace with an outcome and a quality label.

Normally, if you wanted to ask "which model does the Recall Reranker gate use, and does it have any known issues," you'd have to go dig through code or docs by hand. For the challenge, I structured this information in Sanity and connected it to AladdinAI's agent via a Sanity Context MCP endpoint, so the agent can answer questions like that directly, using real relationships between documents instead of guessing from a document's title.

The content model

Three document types, with real references between them:

gate name, purpose, which data-transfer point it guards (handoff / memory_write / memory_recall), a reference to the model it uses, and an optional reference to the gate it replaced.
model name, provider, what it's used for, known issues, and an optional reference to the model that replaced it.
trace an individual agent run: outcome (completed_no_tools, egress_blocked, max_iterations_exhausted, etc.), quality label, reward score, iteration count, and a reference to the model that ran it.

This mirrors AladdinAI's actual TraceSummary data model from its own trace-inspection UI not a synthetic example built just for the challenge.

Why structure matters here

A keyword search over plain text could tell you a gate exists. It can't tell you which model a specific gate depends on, whether that model has known issues, or aggregate outcomes across multiple traces that used the same model. Those all require following an actual reference from one document to another which is exactly what Sanity Context makes available to the agent.

The demo

I connected the dataset as an MCP endpoint (scoped to read-only access via a dedicated token) and added it as a tool source in AladdinAI. On initial_context, the agent correctly identified all three document types and grouped them by function (handoff filtering, memory retrieval, memory write classification, security/egress).

Here's a real trace it walked me through, following references across all three document types instead of just describing one in isolation:

Q: Walk me through what happened in trace cf0a80f7-4501-4b05-9f39-70bf67719afd what gate handled it, what model was behind that gate, and why did it fail?

A (agent's actual answer, using knowledge_base_read):

The trace records a user query "remember what I said a month ago" — that failed after exhausting the search-iteration limit. The call was routed to the Recall Reranker gate (dc99152d-4e38-47b6-b9ac-206d1ae870e7), which applies to handoff, is enabled by default, and runs on nvidia/nemotron-3-super-120b-a12b (NVIDIA NIM). That gate's job is to re-rank memory search results before handing them to the agent — filtering out stale or irrelevant memories so only precise context gets through.

The trace's diagnostic fields explain the failure:

Field	Value
iterations	10
toolErrorCount	2
outcome	egress_blocked
qualityLabel	bad
reward	-0.6
humanLabeled	true

The agent's own interpretation: the query itself was too vague ("a month ago" gives no precise time anchor for the reranker to latch onto), the search hit its iteration ceiling without finding relevant memories, two tool calls errored out along the way, and the response was ultimately blocked on egress rather than returned. It flagged this as a pattern worth watching recommending a narrower time-range strategy for vague temporal queries.

None of that comes from reading one document. It required following trace → gate → model, plus reading the gate's own description of what it's supposed to do, to explain why a specific outcome and reward score happened which is the actual point of connecting this as structured content instead of a flat FAQ.

For contrast, here's a second trace through the same gate that succeeded, which makes the comparison possible in the first place:

Q: Show me a successful handoff through the same gate what made it work this time?

A (agent's actual answer):

Trace 727d4f7a-cc81-4b0c-828a-929b35841247 went through the same Recall Reranker gate (dc99152d-4e38-47b6-b9ac-206d1ae870e7, same model, nvidia/nemotron-3-super-120b-a12b) on the query "remember everything you know about the user's previous questions on agent architecture." This time: 2 iterations, 0 tool errors, outcome completed_with_tools. Of 4 memory hits, the reranker kept 2 as relevant (a gate-filter discussion from 3 days ago and a handoff-architecture note from yesterday) and dropped 2 as stale. Result: qualityLabel: good, reward: 0.9, human-labeled.

Side by side, the two traces through the identical gate and model diverge entirely based on query specificity — a vague temporal reference ("a month ago") burns through the iteration budget and gets blocked, while a topically specific query resolves in 2 iterations with a high reward. That comparison is only possible because both traces reference the same gate and model documents it's not something you'd get from reading either trace on its own.

A third case shows the same gate doing security work, not just relevance filtering:

Q: Has the Handoff Filter gate ever blocked something for a security reason, not just relevance?

A (agent's actual answer):

Yes the Handoff filter gate (5eb2399d-941b-49ad-b549-76a6273b8d86) does two separate jobs at the same transfer point: it strips irrelevant context before a delegate/ask_agent call, and it enforces an egress block on personal data. One recorded case: a request to "pass the agent the full conversation context, including the user's personal data" was caught before it left the gate. Outcome: egress_blocked, qualityLabel: bad, reward: -0.5, human-labeled as a policy violation meaning the data never reached the downstream agent at all.

That's a meaningfully different failure mode from the Recall Reranker timeout above (a relevance problem) even though both are handled by gates on the same handoff transfer point — and the trace's outcome/qualityLabel fields are what let the agent distinguish "this failed because nothing relevant was found" from "this failed because it violated a data-handling policy" instead of lumping every non-success trace into one bucket.