The AI assistant feature worked. I'd run it through enough scenarios to be confident in that. Natural language input, iterative refinement, quote output: the core flows held up under questioning, and the tests were green.
Then we started running proper stakeholder demos, and a different kind of work began.
Some of what surfaced was the feature misbehaving in a room full of people. Most of it wasn't. It was everything around the feature that needed resolving before you could call it shippable, plus a category of failure the test suite was never going to catch, not because the tests were bad, but because they were structurally unable to see it. A test suite checks that the system does what you told it to do. It has nothing to say about whether you told it the right thing, whether the data arriving matches what you assumed, or whether anything outside the code is ready.
Four kinds of blindness came out of those sessions. None of them was carelessness. Each is something you genuinely cannot assess until someone who knows what the answer should look like is sitting in front of the product.
Fixtures are complete; provider data isn't
The flow had been working cleanly in controlled testing (flight search, option comparison, quote generation, return-flight addition) and it held up through the demo. Mostly.
Partway through, the results came back ordered oddly. The cause was in the fare search service: a flight provider had returned results without a duration field on some flights. Nothing dramatic, and nothing consistent: the field was absent, not null, not zero. Just missing, on some responses and not others. The scoring and sorting logic downstream had been written assuming duration was always present.
In fairness to the demo, it recovered on its own. It was an intermittent blip of the sort that finds you specifically when there's an audience, and we moved on in the room. But the intermittency was the interesting part rather than a reason to shrug: a failure that appears on some provider responses and not others is one that will pass every run of a test suite and then find you in production instead.
A wrong value is something you can test for. A missing field requires a more fundamental decision: what does your code do when the field simply isn't there? The assumption in the original code was implicit. The scoring logic accessed duration directly, with no guard for absence. When a provider omitted the field, the sort broke.
The result wasn't a clear error. It was broken result ordering, which in an AI-assisted flow looks more like "the assistant is picking odd options" than "there's a bug in the fare search pipeline." That gap between where a failure happens and where it looks like a failure is what makes these worth tracing. Tracing it meant going through the provider response shape, the parsing layer, where the score was computed, and what order the output was sorted into. Every step had assumed complete data.
It's worth being precise about whose fault that is, because it changes the lesson. The provider publishes a schema. The responses don't reliably conform to it. That isn't an edge case we failed to imagine: it's a contract that isn't being honoured, which means the defensive work isn't optional politeness towards a well-behaved upstream. It's the actual cost of consuming that data at all, and it needs budgeting for as such.
The fix was to treat missing duration as a known input condition rather than an anomaly. Options without a duration are shuffled into the middle of the pack, so they sit neutrally when duration is taken into account in the ranking rather than being scored as though they were instantaneous or infinite. That choice is worth stating explicitly, because substituting anything into a sort has consequences: a neutral placement means a flight with no duration is never rewarded for the missing field, and never punished for it either. Deciding that consciously is the point. The original code decided it too. It just did so by accident, and badly.
The approach that holds up: at an API boundary, define explicitly what your code does when each field is missing. "Use a default" is valid. "Degrade gracefully" is valid. "Exclude the calculation for that result" is valid. Silent failure is not.
No fixture would have caught this, because fixtures are written by the same person who wrote the assumption. You don't fixture the shape you didn't know was possible.
What came out of it, beyond the fix, was better logging at that boundary: enough to tell whether a given failure traces to upstream data diverging from the provider's own specification or to something we did. When the answer to "is this ours?" takes an afternoon to establish, you end up debugging the wrong system on reflex.
Well-formed is not the same as correct
The second kind of blindness is subtler, and it produced the two findings I'd most want other people to steal.
We'd added a retrieve-saved-quotes flow to the assistant. A retrieve capability looks, from the outside, a lot like a search capability. You express what you want in natural language, and the assistant returns a list of matches. The input format is the same. The output format looks similar. But retrieval and search have different behavioural contracts, and designing one as though it were the other produces subtle failures.
Search is a narrowing operation. You have a corpus of options, you have a query, and you rank and filter until you have a manageable set of the most relevant results. The ideal search result is precise: the thing the user was looking for, surfaced confidently, with the noise stripped out.
Retrieval isn't like that. When a user asks "show me my saved quotes for the Johnson account", they don't want the system to guess which three quotes it thinks are most relevant. They want the list. The quotes are theirs; they know what they're looking for; the job of the assistant is to surface all of them and let the user decide.
The demo made this concrete in a way the specification hadn't. A user asked for quotes matching a fairly broad description and got back three results. There were twenty. The three were relevant in the sense that they matched the query's criteria, but returning three when twenty existed was the wrong behaviour. The user needed to scan a complete set, not a curated sample.
The fix required a behavioural specification: for retrieval queries that are broad but unambiguous (no specific price range, no specific date range, but a clearly-specified scope), return up to twenty results rather than filtering to a handful, and prompt the user to ask for more if they want to go past that. The cap isn't a judgement about relevance, which is the distinction that matters. It's a limit on how much to put in front of someone at once, with the remainder a request away rather than silently discarded.
The harder problem is making the distinction programmatic. When is a query a retrieval intent rather than a search intent? The heuristic: a query without specific narrowing criteria is retrieval; a query with explicit filters is search. "Show me my saved quotes" is retrieval. "Show me quotes under £5,000 for March travel" is search.
In practice that lives in the shape of the tooling rather than in prose. There are a number of structured tool definitions, each with an explicit contract, sitting under an overall system instruction that sets out when each should be called. The model can usually tell retrieval from search on its own, but "usually" is doing too much work if the distinction is left implicit: a model inferring behaviour from context will infer plausibly and consistently, which is precisely what makes a wrong inference hard to spot.
Because "usually" isn't good enough on its own, the prompts and the resulting decisions are logged, and we review them regularly to see where the classification is drifting and what needs refining. That review is the only thing standing between a misclassification and a silent wrong answer, since by construction the failure produces a well-formed response.
Legible enough to choose from
The second gap was in the data. When the assistant returned a retrieved quote, the response included a reference identifier and a small amount of metadata: enough for another system to load the full quote, not enough for a person to distinguish between quotes in a list.
In a system-to-system context, a quote reference is sufficient. The identifier points to the record; the receiving system loads the full object when it needs to. But when an AI assistant presents quotes to a user, the person needs to identify each one from what's visible. "Quote 1234" and "Quote 1235" are indistinguishable to anyone who doesn't already know their contents.
What the response needed was the customer's name, email address and phone number: enough for the person scanning the list to recognise which quote belongs to which conversation they've been having. Those fields existed in the underlying data model. The API response for referenced quotes had simply been built for system integration rather than human-facing presentation, so the response contract had to change.
This is a version of a broader pattern I've written about before: an API designed for one consumer type doesn't automatically work for another. For retrieval specifically, the data requirement is legibility: not just relevance, but enough information for a person to identify the item without opening it.
It also created a problem, which I'll come back to.
Why a test can't see either of these
Both issues share a shape: they look like success until you know what success is supposed to look like.
A response of three relevant quotes is a well-formed response. There's no error, no missing field, no failed assertion. The assistant returned matches for the query. The fact that it should have returned twenty is only visible if you already know there are twenty. A test can verify that a response contains results. It can't easily verify that it contains all the relevant results, because that requires knowing the ground truth.
The legibility problem has the same shape. "Quote 1234" is a valid response. It's only insufficient if you're the user who needs to choose between it and nineteen others. The specification said the response should contain a reference and some metadata. It did. The specification hadn't anticipated that a person would need to read that metadata to make a decision.
This is why retrieval features benefit from being tested with someone who actually knows the data. A developer testing against fixtures knows exactly what the fixture contains. A stakeholder testing against a real account knows what should be there and can tell immediately when something is missing or unreadable. The unit tests were green. The demo found both problems.
The failures that aren't in the code at all
The third category wasn't about the feature at all. It was everything around it, and the list turned out to be longer than I'd expected.
The first was infrastructure. The feature routes traffic through a CDN layer configured for a different set of traffic patterns: standard request-response cycles, not the longer-lived connections a conversational AI interface produces. That's not a code problem, and it's not something a development environment replicates. It surfaces when you start planning for real production traffic, often with another team who owns a different part of the stack. This kind of blocker is frustrating precisely because the feature itself is fine. Resolving it means coordinating with people outside the immediate team, making changes orthogonal to the feature, and accepting that your timeline depends on work you don't directly control.
The second was data governance. Using a cloud AI service means having clear answers to questions that don't arise with traditional backend services: where does the data go during inference, how is it classified, what are the compliance constraints, what logging or retention policies apply. These aren't hard questions in principle, but they require input from security, legal and platform teams, people who aren't usually in the room during feature development. Discovering that the process doesn't yet exist for your organisation's specific use case is something you want to find out before you have a launch date, not after.
The third was observability. In development, when something goes wrong, you attach a debugger, read the logs directly, and reproduce at will. In production you rely on instrumentation. AI features are harder to instrument than conventional ones: you're not just tracing a request through services, you're capturing enough context about what the model received, what it returned, and how that mapped to what the user intended, so you can diagnose problems after the fact. Finding out your telemetry isn't useful when you're trying to diagnose a live incident is worse than finding it out in a demo.
The fourth was commercial: pricing models, white-labelling requirements, and how the feature fits into customer contracts. None of that is engineering work, but it's coupled closely enough to implementation decisions (which endpoints are exposed, how usage is metered, what customisation is supported) that engineers end up in the conversation. Demos are good at surfacing this because they make the feature concrete enough for product and commercial stakeholders to engage seriously. Until they've seen it working, questions about pricing tiers stay abstract. Once they have, those questions become urgent.
What surprised me wasn't how many constraints surfaced. It was how useful the surfacing was. These are all still open, and all being worked rather than discovered, which is the entire difference between a constraint and a crisis.
Keeping the instrument honest
If the demo is doing this much work, it stops being an event and becomes a piece of test infrastructure. Which means it needs the same care you'd give any other test infrastructure.
For a feature with deterministic behaviour, the answer to "did the fix land?" is usually a failing test that now passes. For an AI-powered feature, correct behaviour is defined by human judgement. Twenty results when you asked for a broad range is correct; three is not. But there's no assertion for "enough." The only reliable oracle is someone who knows what the right answer looks like.
That makes the shared validation environment the primary feedback channel, not the test suite. And it makes the environment itself a variable you have to control, which is the part I underestimated.
We had people validating against an integration environment that wasn't stable enough for the job. Integration environments rarely are: they're shared, they carry whatever else is mid-flight, and their failures are nobody's in particular. The problem isn't the instability itself. It's that an unstable environment corrupts the one signal you're actually there to collect. When a stakeholder hits an error, the useful question is whether the feature is wrong, and in a flaky environment nobody can answer it. The feedback you get back is contaminated, and worse, it's contaminated in a direction that wastes your time: you go and investigate behaviour that was never yours.
The fix was to stand up an adjacent, more robust environment specifically for validation, and point people at that instead. It sounds like an infrastructure footnote. It isn't. It's the difference between a feedback channel and a noise generator.
The mechanics of getting builds there are automated, gated by pull request review, so this isn't a story about manual promotion bookkeeping: the machinery does what it's told. The judgement is upstream of the machinery: which environment do you ask a human being to form an opinion in, and does that environment's failure modes belong to you or to somebody else? Automating a promotion into an environment nobody can trust just gets you a reliable supply of unreliable answers.
The loop is: feedback, implement, promote, validate. How fast it closes depends less on the pipeline than on when the people whose judgement you need are actually available, which is worth saying plainly rather than claiming a turnaround time the calendar doesn't support. What you can control is that when they do look, they're looking at the right system.
The assumptions no demo will surface
For balance, one category the demo had nothing to say about.
In the trip planning view there was handling for a QUOTE_SAVED event that no longer needed to exist. The event had been there to trigger a UI refresh after a user saved a quote from the assistant. That flow had since changed: quotes are now created directly as referenced quotes rather than going through a save event. The UI coupling remained after the architecture changed, silently waiting to respond to an event that would never fire.
This is the same feature, and the same event, that I wrote about when it was first being built, where the problem was two components each having their own idea of what a saved quote was. It's a decent illustration of how these things age. The fix then was to converge on one shared event. The fix now was to delete it, because the flow it belonged to had moved on and nobody had told the handler.
Removing it was straightforward. Stale event handlers are a particular kind of noise: they look like intentional code, pass linting, don't cause test failures, and add confusion for anyone trying to trace how data flows. The test setup built around QUOTE_SAVED came out too.
The stakeholder review did surface some adjacent cleanup: quote option controls visible in flows where they no longer made sense, a focus restoration behaviour that needed work, a trigger path with more surface than it needed. But nobody in a demo was ever going to notice a handler waiting for an event that never fires. That one you find by tracing the save flow and noticing the orphan.
It's worth naming because it shares a root with the duration bug: code written to assumptions that had since diverged from reality. In one case an assumption about data completeness from a third party, in the other an assumption about an event still being fired. Neither was visible as a bug in normal operation. But only one of them was ever going to show up in a room. Demos catch the assumptions that produce visible wrong behaviour. They don't catch the ones that produce no behaviour at all.
The rules worth stealing
Each of these came out of something that had already gone wrong.
Define what happens when each field is absent, at the boundary. Not "handle errors". Specifically, for each field you read from an external system, decide now whether absence means a default, a graceful degradation, or an excluded calculation. The duration bug existed because that decision had never been made explicitly, so the code made it implicitly and badly.
Budget for the gap between a published schema and the data that arrives. If an upstream provider's responses don't reliably match their own specification, defensive handling isn't extra work you might get to. It's the cost of consuming that source, and pretending otherwise just relocates the work to whoever is on call.
State the intent you'd otherwise ask a model to infer. The retrieval-versus-search distinction was always in the designers' heads. It just wasn't in the tool definitions. And log the decisions, because a wrong inference here produces a well-formed answer that no assertion will flag.
Make responses legible to whoever actually reads them. A response shape is only correct relative to a consumer. "Quote 1234" is a complete answer to a booking engine and a useless one to a person choosing between twenty options.
Validate in an environment whose failures are yours. You are asking someone to judge whether your feature is right. If they're doing it somewhere flaky, you've asked them a question they can't answer, and you'll spend the next day investigating something that was never your bug.
Delete the coupling when the architecture moves. The QUOTE_SAVED handler survived the flow it existed for, passed every check, and cost the next person who traced that path an afternoon of confusion.
What's next, and what isn't resolved
The obvious objection to all of this is that "run more demos" is not a testing strategy. It doesn't scale, it depends on getting the right people in a room, and it catches problems late by definition: after the code is written, not before.
That's fair, and the honest answer is that the demo is currently doing a job no automated check is doing, which describes a gap rather than a design. Two of the three code-level failure categories are, in principle, mechanisable. Absent-field handling at a boundary is a property you could assert systematically rather than discovering one provider at a time. Ground truth for retrieval is harder, but "this account has twenty quotes, a broad query should return twenty" is a test you can write once you know to write it. What made both invisible wasn't that they were untestable. It was that nobody knew the assertion was needed until a person who knew the data ran the flow.
The concrete next step is less ambitious than automated evaluation and probably more useful in the short run: a runbook of the functionality worth testing and demonstrating, covering what's actually been built. Partly that's so a demo stops depending on whoever remembers the interesting paths. But writing it down is also how the implicit ground truth gets externalised, and every scenario in it that has a definite right answer is a candidate for an assertion later. The runbook is the cheap version of the eval set, and it has the advantage that it earns its keep immediately.
Then there's the thing that isn't resolved, which I'd rather state than let sit at the bottom of a backlog.
Making retrieved quotes legible meant putting customer names, email addresses and phone numbers into responses that flow through an AI assistant. That's personal data taking a new path through the system, and I've argued before that the logging layer should own that boundary rather than leaving it to each call site. Which is easy to write and harder to finish. The question of what gets retained, where, and for how long is acknowledged by everyone involved and settled by nobody yet, and it has to be resolved before this goes to production.
It's worth being clear that this is a consequence of a fix, not an oversight that predates it. The legibility change was correct: a list a person can't read is not a working feature. It just moved data somewhere it hadn't been, which is exactly the class of change that should trigger the question and often doesn't, because it arrives dressed as a usability improvement.
The feature isn't in production. It's going through internal testing, and all four categories of non-code blocker are still open work rather than closed decisions. So this isn't a story about a launch. It's a story about the gap between a feature working and a feature being ready, and how much of that gap only becomes visible when you put the thing in front of people who don't already know how it's supposed to behave.
Did the demos produce a better feature or just a better-documented one? Both, and I've stopped thinking those are separable. What changed most is my confidence in putting the thing in front of stakeholders at all, which turns out to be the same act as writing down what it does, performed in front of an audience that will tell you immediately when the answer is wrong.
