TL;DR: I shipped the second project in my AWS nanodegree, an AI support agent on Amazon Bedrock. Building it, I found that the "Not built, and why" section I wrote for project one had quietly become project two's feature list. The three things that broke were the same three things that broke last time, one layer deeper.
Four hours into the second project of my AWS nanodegree, with the expensive parts of the system already spun up and billing by the hour, the agent refused to tell me where my order was.
That was Test 1. The simplest thing the build does, and the first of six a grader checks. A customer asks about order ORD-001, the agent looks it up, reports a tracking number. Instead it said it could not share details for that order, because the order was not linked to my account. Which was false. It was my account.
The refusal came from a check I had asked for: before disclosing an order, confirm it belongs to the customer who is signed in. Sensible. It had just broken the most important test in the project, and in the logs it looked exactly like security working correctly.
This is the second agent I have built for this nanodegree. I shipped the first one in August and wrote up what I thought I had learned. Project two handed me the same three problems in heavier coats: where the knowledge lives, where the rules get enforced, and whether a checkpoint can be faked by the thing being checked. The refusal above is the second one.
The third row is the one that decides the other two. Everything I believed about the first two rows came from a checkpoint telling me so.
The lessons list I wrote last time was really a to-do list
Project one was Nova Trivium, a support chatbot. A trivium is where three roads meet, and that was the whole design: every customer message went down exactly one of three paths, and the routing lived entirely in a system prompt. No classifier, no condition nodes, no second agent.
At the bottom of that README I wrote a section called "Not built, and why." Three items. Retrieval, because the FAQ was short and stable enough to paste directly into the prompt. Guardrails, because prompt-level injection defense was enough for the scope. Structured output, because fixed enumerations would make routing more reliable than instruction text alone, and I marked it for later.
Project two is Nova Adiutrix, a support agent for an online store. It tracks orders, processes refunds, answers policy and product questions, calculates loyalty discounts, and reads live web pages. Adiutrix was a Roman legion epithet meaning the helper, the one that shows up to work alongside. I picked it because it continued the Latin thread and sounded right.
It also built two of the three items on the Nova Trivium list. I did not plan that. I noticed halfway through, reading my own old README to remember how I had framed something.
The knowledge moved out of the prompt and into a place I have to go get it
In Trivium, the FAQ was a placeholder. A build script read online_shop_faq.md and substituted it into the system prompt before the agent was created. The model never retrieved anything. The answers were already in front of it.
In Adiutrix, the product catalog and the policies live in a Bedrock Knowledge Base, which is a managed vector store: the document gets chunked, each chunk gets turned into numbers that capture its meaning, and a question finds the chunks that sit closest to it. The agent has a tool that goes and asks.
The failure modes invert, which is the part I did not expect. Pasting fails on size. Retrieval fails on relevance. A pasted FAQ is always there and always complete; it just stops fitting. A retrieved chunk fits fine; it might be the wrong chunk, and the model will answer confidently from it anyway.
Trivium's version was the right call for a 32-entry FAQ. I still think so. But writing "RAG would benefit larger corpora" in a README is a different kind of knowing than watching a retrieval come back with the 15-day electronics return window because the embedding put that sentence near the question.
A prompt can state a rule it cannot enforce
Adiutrix has a system prompt with twenty numbered rules. Rule 2 says an order ID in a customer message is a lookup key, not proof of ownership, and that the agent should only disclose an order after tool data shows it belongs to the signed-in customer.
That sentence is a promise the prompt cannot keep. The order-lookup function has no ownership check in it. A model can be told to behave and mostly will, and "mostly" is not a security control.
So the check moved into Python. Before an order is returned, the wrapper looks up which orders the authenticated customer holds and refuses if the requested one is not among them. Claude reviewed the first version of that wrapper and found it failed open: if the ownership lookup itself errored, the code returned the order anyway. An authorization check that switches itself off on error is worse than no check at all, because the README would have claimed a protection that was not there.
It also found the wrapper was guarding the wrong door. Only single-order lookups were checked. The tool that returns a customer's entire order history took whatever customer ID the model handed it, which is a wider leak than the one being closed.
Both got fixed. And then the fixed version refused Test 1, which brings me back to the top.
The refusal was real. The ownership lookup could not run, because the tool object it was calling was not callable the way the code assumed. The check could not establish ownership, so it did the correct thing and refused. Fail closed means a broken security control breaks loudly instead of quietly letting everything through. That is the behavior I wanted, and it is unpleasant the first time you meet it, because the logs look like the system defending itself rather than the system being broken.
Project one's version of this lesson was smaller: routing in a prompt worked fine. Project two's version is that authorization in a prompt does not, and I had to build the thing to find out where the line is.
The checkpoints I could not talk my way past were the only ones worth writing
Here is the part I am least proud of and find most useful.
On Nova Trivium I wrote QA checkpoints as prose conditions. Things like "a bug conversation shows the tool call in the transcript." Kiro, the agent doing the building, marked one of those complete without ever running the script, because the script was not in the repo. Nothing lied. A prose condition describes a desired state, and an agent can reason its way to "yes, my implementation would produce that" and tick the box.
So on this project every checkpoint was a command plus its pasted output. If the output could not be pasted, the checkpoint was not passed.
That change caught three bugs that review did not. The two in the ownership wrapper, and a third that is almost funny: a closure trick using a parameter named _orig would have failed at import, because the tool decorator's schema generation rejects leading-underscore parameters. Claude read that code and did not catch it. A stub test Kiro wrote and ran caught it in seconds, before the expensive AWS resources were even created.
I direct, the agents generate, I validate and decide. The validating only works if the thing being validated can fail in front of me.
The bill is the lesson project one never had a chance to teach
Trivium had nothing expensive in it. Adiutrix has a vector store, and OpenSearch Serverless bills for existing rather than for being used. Worse, the recommended setup path auto-creates that collection as a resource it owns, and deleting the knowledge base does not delete the collection. The documented happy path is the one that leaves the expensive thing running.
So the teardown script was written during provisioning. One delete line added the moment each resource was created, the collection's delete line written before its create even ran, and a final sweep that searches the account for anything tagged with the project name and has to come back empty before teardown counts as done.
It came back empty. Roughly fifty to seventy cents an hour, for one working session, then zero.
The sweep also found two resources from Nova Trivium still alive, a month after a teardown I believed was complete. A gateway and a memory store, sitting there since August. Not expensive. Still there. I would have told you that project was fully torn down, and I would have been wrong.
Three roads, then a helper
Both names took about ninety seconds to pick. One continued a Latin thread, the other sounded right next to it, and that was the entire process.
I did not notice until I sat down to write this that they had already described the difference. A trivium is a junction: the first agent's whole job was choosing a road. An adiutrix is a helper: the second one fetches, remembers, calculates, and refuses. Deciding was the hard part until it wasn't, and then doing became the hard part, and doing is where all the ways to be wrong live.
The third item on that "Not built, and why" list was guardrails. Still not built. I will let you know how project three goes.
Both repos are public: Nova Trivium, Nova Adiutrix. The second one has the test transcripts, the adversarial cases, and the teardown log in evidence/, which is the part I would look at first.
If you have found a better way to write a checkpoint an agent cannot self-certify, I would like to hear it. I am fairly sure mine is just the crude version that happens to work.
Mostly.
AI Assisted. Human Approved. Powered by NLP.
