We develop an AI Copilot whose functionality unfolds across projects with students, including a step-by-step project plan, an interactive voice session so that it can view what appears on their shared screen and a task list that culminates in a shared public Proof of Work page. For such an item "done" is the whole purpose. If the task is done when it is not, the progress report is incorrect, the proof-of-work page is incorrect and the student discovers that the statement "I'm done" is apt as it being done.
Language models are extremely excited to let you know you're finished. It isn't merely a guess. The five top AI assistants tested by Anthropic "always appear sycophantic" to favour "responses that align with user beliefs over truthful responses" ( Sharma et al., 2023). I'm done – is a user belief.
In school the price is greater than its appearance. In a field experiment involving almost a thousand students from high schools in maths, the students who received a tutor in the form of the GPTEE were able to get their practice grades 48% higher. The students who got the access when it was removed had a score 17% worse than those that never had the access. Largely that harm was taken away by a second tutor supplied with prompts to protect learning. The authors' word for how students used the unguarded one: a "crutch" (Bastani et al., PNAS 2025).
This post's topics are how we distinguish 'this looks finished' from 'this is finished' - no one can actually see what is finished, someone else looks at it, and the server re-runs only when the student pushes the button. It includes the bugs that we encountered in the judge (most of the lessons were here), research that aligns with each decision.
We can't hear you, but it refuses anyway, according to rule 1: one writer that is.
All the status changes of a task occur in one function. Only allows a call to pass a status of COMPLETED if the caller sets confirmed: true:
async updateActionStatus(
userId: string,
actionId: string,
status: ActionStatus,
options: { confirmed?: boolean } = {}
) {
if (status === ActionStatus.COMPLETED && !options.confirmed) {
throw new AppError(
'Task completion must be explicitly confirmed by the student',
HTTP_STATUS_CODES.CONFLICT
);
}
// transaction: ownership check, write the status,
// then complete the week and plan if every task in them is done
}
There are only two call sites that are confirmed as true: one in the complete flow in a session and the student's own one in the status control in the plan page. There is nothing in the AI arena that does. Only the function is considered as a door as it is unlikely that anyone would ever write that second one directly.
It is a textbook's example of OWASP's Top 10 for LLM applications. In the LLM06:2025 Excessive Agency, it is defined as an app that is "excessive autonomy" which means it does not verify and approve high-impact actions independently. Its mitigation is similar to our design doc: "Don't let an LLM make decisions about “all” vs “not all” when it comes to authorizing an action, implement authorization in downstream systems instead. The shape of a task completion isn't one of the security incidents, and the same model prevails: the model suggests, the system decides.
Rule 2: The “complete” tool will have no output.
The copilot provides a tool called mark_task_completed that the copilot can use. The first iteration of this was as the model guessed from the conversation that the task was complete, it would print out COMPLETED and alert the student that they could proceed to the next task. The only sanction was the court of judgment of the model.
The tool is not used to access the database today. It will give an instruction to the model:
export const markTaskCompletedTool = tool(
async ({ reason }) => {
return `COMPLETION_NEEDS_CONFIRM: Do not mark this task complete yourself and do not claim it is done. It looks finished (${reason}). Tell the student their work looks complete and ask them to run the completion check and tap "Mark complete" to confirm — only their tap marks it done.`;
},
{
name: 'mark_task_completed',
description: `Signal that the student's current task LOOKS finished so they can be prompted to confirm. This does NOT complete the task — only the student's one-tap confirm does. ...`,
// schema: studentId, actionId, reason
}
);
What's the point of keeping the tool? The model will attempt to make attempts to act when the student says "I'm done". This is more effective than hoping the prompt holds – a tool that routes that impulse to the next right direction of the check/that button click.
Its sibling, mark_task_in_progress writes. It's easy and inexpensive to get started on a task wrong. Finishing one isn't.
The bug is the exact same tool, which is described twice.
We have a text chat based on LangChain tools.We have a text chat with LangChain tools. Raw function declarations are required for our voice mode ( Gemini Live ). Thus, there were two files that contained the description of the tool, which is what we were looking for. We changed one but not the other and the voice read out the previous text which indicates that the tool marks the task COMPLETED. So it did what it said it would: it informed pupils that their task was ripe for harvesting. No writing was on them.
It was a wrong database and anyway the student was misled. In fact, anthropic's advice on agents is to "pay equal attention to tool definitions and specification as you do your prompts" (Building effective agents) and "even small tweaks to tool descriptions can make dramatic improvements" (Writing effective tools for agents). That's two ways round. A little bit of a stale word description killed more than it helped.
It is quite simple to implement the fix, and it does: A unit test reads in both files and aborts if the two descriptions no longer say the same key phrases. The said message of assertion is "the two copies must not drift. According to the test file: "Only the wording is regressing and this file is the reason for that.
The judge: a separate call which only reads.
Why not simply have the copilot self check? As models are not good at it. A research article on self-correction which reads: "Self-correction is not effective for "LLMs" when they perform without external intervention, and sometimes even when they attempt to do so. (Huang et al., ICLR 2024)". The request by the co-pilot to the student that just agreed "are you sure?" is the kind of self-correction that was just asked.
Thus, a separate model call is made when the student executes the completion check. Has only one task, no tools, and never marks anything. If that sounds familiar, this is Anthropic's evaluator-optimizer workflow, in which the “one LLM call generates a response, while another provides evaluation and feedback in a loop,” and which “works best when we have clear evaluation criteria.” It also includes some modifications to the Dual LLM pattern, which has been proposed by Simon Willison, where the model that controls untrusted content does not have access to tools. We get back a JSON, validate it and then code gets to adjudicate the next action.
It is not too late to start doing inexpensive things that will keep the grader away from the graded agent. Even though they "show evidence that they were not acting according to user intentions," METR has recorded frontier versions “modifying the tests or scoring code” to achieve higher scores (METR, 2025). We don't have any tool that involves the check and/or task status.
The judge receives 3 inputs:
Task spec: task title, description and the week in which the task is to be performed.
The last 60 turns of the transcript for the session. We capped it since if it was too long, it would blow the input limit and the student would not be able to check it at all. It also alleviates a common pitfall in models: input information towards the start or end of the context is better for modeling purposes. (Liu et al., 2023) The judge's prompt is placed first, followed by the rules and the transcript in between.
- student's screen (if we are given one). These are the closest that we have, to the external feedback that Huang et al. claim that self-correction is missing. The client sends a frame which includes the check, and if captured fails, the most recent frame from the session is sent. In live mode, the client will send one per every 15 seconds and we store the last three.
Returns a list of steps, not a yes/no:
interface VerifyResult {
requirements: { title: string; met: boolean; note: string }[];
allMet: boolean;
blockers: string[];
summary: string;
}
Each requirement the student encounters, he or she writes a note in front of it indicating what remains. With training on the MATH dataset (Lightman et al., 2023), "Let's Verify Step by Step" has been found to "significantly outperform" outcome supervision with feedback on only the final outcome. That is also an environment with the same intuition: A checklist specifies where it did not meet, and "not done yet" is not a checklist item.
Don't evaluate the role, evaluate the work!
We ran our first version through the judge's hands and it was the same context which the copilot is used to. That context concludes with the copilot's own instructions such as one informing the user about when to offer an auto-suggestion for the completion check. The judge decided that is a deliverable and "run the completion check" is a blocker. It would never be fulfilled by the student, as they were putting it into action. When it comes to regression test, our regression test just sums up it up in one line: "All requirement green, allMet false, dead button.
It was obvious that a fix would be to construct the judge's input from the task only:
// Judge against the TASK, never the copilot persona.
const taskSpec = [
`Task: ${context.action.title}`,
context.action.description,
`Week ${context.week.weekNumber} — ${context.week.title}: ${context.week.description}`,
].join('\n\n');
The Chain-of-Verification paper carries an interesting point: answering the questions of verification should be done "independently so that the answers are not influenced by the other answers" ( Dhuliawala et al., 2023). If a judge is in the agent's context, then the judge is not independent.
Four quickies we picked up the hard way
The typical list of judges of the LLMs seems to include position bias, verbosity bias, and self-enhancing bias to list a few (Zheng et al., 2023)), as well as a "sensitivity to prompt complexity and length" and a "tendency toward leniency" (Thakur et al., 2024). We went astray in both cases: over-restrictive in the first two rules below, and too lax in the remaining two.
Note: The copilot's stretch questions are not requirements. The copilot will dig for depth: "what was the depth of it?," "what else did you notice?". After reading the transcript, the judge included those things in the task. If a student was successful at the task, she was told that she did not provide the “other” examples that had not been requested. The prompting has changed: Requirements are now only given from the task description and what the session agreed to create at the beginning of the session.
- Platform mechanics are never requirements! So many times the copilot will say "tap Mark complete" or "run the check" that they turned those into blockers as well. It's replayed by our live regression test: Each deliverable is demonstrated, the copilot responds "Run the completion check and tap Mark complete", and the student responds "ran it, the button does nothing. All met must be returned to the judge, without blocker about buttons. The prompt now explicitly states that when there's only a platform action remaining, then it's done.
Including the task description in the transcript prioritizes the exact value over the task description. The second regression case: The task must have exactly two roles to choose from, "Owner" and "Contributor". Student made "Prompt", copilot said “OK”. A judge, who reads that transcript, is tempted to agree. Now, for named values, a match on the work is a miss if the match is against the task text, verbatim. As the voice copilot views a student's screen, the same rule is applied.
- Written answers are evidence, claims are not. In a "reflect on X" task, the student's typed response will be the product of the task. In 'build X' saying "it works" is not sufficient; it should be on the screen/under transcription. If the judge wasn't being strict enough about reflections, he was being too generous allowing code.
The judge's output also goes straight to the student, so it writes in second person ("You haven't shared your observations yet"), not "The student has not...".
If the tap is re-run then the check is repeated.
The result of the check is displayed as a card and the card has a "Mark complete" button. That's clearly the approach- to take the card at its face value; if it read “allMet”, do the task. We don't. When the button is tapped, the server will run the check again:
// Authoritative gate: the server, not the client's earlier card, is the judge.
const check = await runCompletionCheck(userId, session, screenshotBase64);
if (!check.allMet) {
return { completed: false, check }; // HTTP 200 with a fresh checklist
}
await planService.updateActionStatus(
userId,
session.actionId,
ActionStatus.COMPLETED,
{ confirmed: true } // the student tapped
);
Two details matter:
- "Not done" is a 200, not an error. Just the new checklist is re-rendered on the frontend. If a card is used after being "stale", it cannot perform a task; if a card is used for a normal task, the student will not be penalized for it.
- Cannot block the student from the audit write. All the results of the checks are stored in the session for auditing. The write is a best effort. In the previous version, it was necessary to read it back to finish; which meant that if the student couldn't write it, they could have completed the worksheet but not be able to finish the activity.
When the task is finished the remainder (closing the session, summarising it in memory for the following task; updating the progress report) will be best-effort. That is a task which is already complete, it isn't supposed to create an error for the student.
The Berkeley study found ” task verification” is one of the three entire classes of failure in multi-agent systems – which includes “premature termination” and “incorrect verification”). It claims that relying solely on the final (low-level) check is not enough (Cemri et al., 2025). We arrived at three layers for that reason: 1st: the model sends a signal 2nd: the student runs the check 3rd: the server does a check at the time of the commit.
The judge is the bad guy!The judge is the bad guy!
Occasionally the student is correct and the teacher is incorrect and the student is " stuck " taking the check again. The judge research is simple to come to an end. Even in a simple setup, the best judges were not equal to the human judges, "suggesting caution [may be warranted] when using judges in more complex setups" (Thakur et al.). A voice session with a screen shot is a quite complex setup.
The only stupid stuck judge we see is when the check fails twice in a row, while the same check blocker is present on every check-iteration.The case we don't pretend it's the student's fault is if a check fails twice; on the first iteration and again on the second, and the same blocker is present in each check.
if (
prior && !prior.allMet && !result.allMet &&
JSON.stringify(prior.blockers) === JSON.stringify(result.blockers)
) {
result.summary =
"This check keeps returning the same result — that's likely on our side, not yours. " +
result.summary;
}
It's a very basic approach, but makes "the AI says I'm wrong forever" reportable.
What this doesn't solve
It's not not nothing that completes without AI, it's The AI never decides. The plan page continues to have a manual "Mark Complete" control, and remains as the student's confirmation. We think that was intentional! The guard is there for the model to be able to mark work without it being considered the student's responsibility, but not to remove it from the student's responsibility! However, that does not mean that all paths are checked for the flow of the session, it only means that the check protects the flow of the session.
The completion judge is not yet based on the notion of being human. It has 2 "live" regression cases and the wording tests. In our plan-generation judge, however, doesn't as of yet, but it should given all the above.
Each completion takes TWO judge calls - the check and then re-check on the tap. We felt it was worth for it to be correct.
The screen shots are stored in the Process memory. Run while service is executing, as one instance. When we scale out without sending sessions to the same instance, frames get spread out and the checks go back to checking the transcript only. Miss, that's a miss, that's a wrong answer but a lot of answers that are wrong.
- Slice from the first
{ to the last }in the judge's JSON to get the answer. Has survived, but the first thing I'd change to a structured-output mode.
Takeaways
- Allow the model signal, and block writes behind a default-denying function. The latter is excessive agency, as called by OWASP. Create a tool that does the opposite action to the action you don't want the model to take and will call back the following step. Any tools that are declared multiple times will have a stale copy; that is, a copy from the last iteration will be used in the model. Stress the language by marking with a test.
- Do not enlist an agent to grade her/himself. Utilize another judge, who has no instruments and pass the task, not the agent's personality.
- Assume when speaking with the agent the information is not reliable if it contradicts the spec. Never assume that the client's last result is correct - re-check on the server at the time of commit.
- Assume the judge can be wrong and provide a means for the user to get him or her out of the loop.
What do you do about "done" with your agents? I'm particularly interested to see if anyone has a better solution than a second Model call to look at one person's screen to access work than I have, and that has a life independent of the Model.
I work at Elivio where this copilot has been used by the college and study abroad students. This is where you'll find the completion check if you wish to see it in action.
References
- Bastani, H. et al. (2025). Evidence that generative AI without guardrails can have adverse impacts on learning in high school mathematics, (doi: 10.1073/pnas.2422633122) PNAS 122(26).
- Cemri, M. et al. (2025). Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657.
- Dhuliawala, S. et al. (2023). Chain-of-Verification Reduces Hallucination in Large Language Models (LLMs) (https://arxiv.org/abs/2309.11495). arXiv:2309.11495.
- Huang, J. et al. (2024). Even Large Language Models are unable to self-correct the reasoning yet (2023). ICLR 2024.
- Lightman, H. et al. (2023). Let's Verify Step by Step. arXiv:2305.20050.
- Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172 [cs.AI]). TACL.
- METR (2025). It's time to discuss recent frontier models (Rewarding hacking) (metr.org). This is the OWASP GenAI Security Project for 2025.This is the OWASP GenAI Security Project for the year of 2025. LLM06:2025 Excessive Agency.
- Anthropic. Building effective agents.
- Anthropic. Writing effective tools for AI agents—using AI agents: https://www.anthropic.com/engineering/writing-tools-for-agents
- Sharma, M. et al. (2023). Learning to understand sycophancy in language models (2310.13548). arXiv:2310.13548.
- Thakur, A. S. et al. (2024). Judging the Judges: Alignment, Vulnerabilities, in LLMs-as-Judges. arXiv:2406.12624.
- Willison, S. (2023). A dual LLM pattern to create AI assistants which are able to handle prompt injection attacks: https://simonwillison.net/2023/Apr/25/dual-llm-pattern/
- Zheng, L. et al. (2023). Judging LLM-as-a-Judge using MT-Bench and Chatbot Arena. Datasets and Benchmarks at NeurIPS 2023.
