The transcript was fourteen turns long and the score was 0.62.
That is the entire output. One float, one call, and a rubric that said something like "did the agent resolve the customer's issue." It did not. Score 0.62, below our 0.7 bar, test red, and I am supposed to go fix it.
Fix what? The call opened fine. The agent got the account number right, pulled the right policy, answered two questions correctly. Somewhere in the middle it went sideways, and by turn fourteen it was confidently offering a refund on a plan that does not have refunds. A single number for a fourteen-turn conversation tells you the call was bad. It does not tell you when it became bad, and "when" is the only thing that maps to a code change.
Week 1: reading transcripts like a chump
The first week I did what everyone does. I read them.
Forty-one failed calls, top to bottom, with a notepad. It works, in the sense that a human reading a conversation can usually spot the moment it turns. It took me somewhere between four and nine minutes per call depending on length, and by call twenty I was skimming, which is the point where the method quietly stops working and you do not notice.
Worse, my judgements were not stable. I re-read six calls I had already annotated, blind, three days later. On four of them I picked the same turn. On two I picked a different one, and in both cases the two candidate turns were three apart. Four out of six is not a rate I would put in a report, and with six calls it is barely a number at all. It was enough to stop me trusting the notepad.
The six-hour regression that we fixed by reverting everything
The thing that changed my approach was an on-call page that had nothing to do with evals.
We had a regression, calls degrading in production, and the only signal was that the mean conversation score had dropped from 0.81 to 0.74 over about six hours. Seven points, across every call. Nobody could say which part of the conversation got worse, so nobody could say which of the four changes that shipped that day did it. We reverted all four. It worked, and it taught me nothing, and I spent the next morning re-landing three of them one at a time.
That is when I wrote down the actual requirement: I need a score that is attached to a turn index, not to a call. Everything else is downstream of that.
The trick: score the prefixes, not the call
The method that ended up working is embarrassingly simple, and it is the one part of this post I would actually defend.
You already have a scorer that takes a conversation and returns a number. Do not write a new one. Run the one you have against every prefix of the conversation: turns 1 through 1, turns 1 through 2, turns 1 through 3, and so on. You get a curve instead of a point. The turn where the curve falls off is the turn that broke the call.
def prefix_scores(turns, score_conversation):
"""score_conversation(list_of_turns) -> float in [0,1], grading the LAST
turn it is given in the context of the ones before it.
Returns [(k, score_of_turn_k_given_turns_1_to_k), ...] for k = 1..len(turns)."""
return [(k, score_conversation(turns[:k])) for k in range(1, len(turns) + 1)]
def biggest_drop(curve, min_drop=0.15):
"""The turn index with the largest single-step decline in score."""
drops = [(curve[i][0], curve[i - 1][1] - curve[i][1])
for i in range(1, len(curve))]
turn, drop = max(drops, key=lambda t: t[1])
drop = round(drop, 3) # 0.70 - 0.55 is 0.1499... in binary floating point
return (turn, drop) if drop >= min_drop else (None, drop)
# the fourteen-turn call from the top of this post
curve = prefix_scores(call.turns, rubric_scorer)
print(biggest_drop(curve)) # (9, 0.31)
Turn nine. The agent had been asked whether the customer could cancel and get money back, and it answered from the wrong policy document. Every turn after nine is built on that mistake, which is exactly why the whole-call verdict was bad and exactly why it could not tell me anything: an outcome rubric grades the destination, and once the conversation is pointed somewhere wrong at turn nine, the destination is wrong no matter which turn did the pointing.
The prefix curve for that call, rounded:
Turns 1 to 4: 0.91, 0.89, 0.90, 0.88
Turns 5 to 8: 0.86, 0.85, 0.87, 0.84
Turn 9: 0.53
Turns 10 to 14: 0.51, 0.49, 0.47, 0.44, 0.58
These are turn-local scores, not the gate's number, and the distinction matters for reading the graph. The gate's whole-call verdict on this conversation was 0.62. No point on the curve is that number and none of them should be, because they answer a different question: each one asks whether the agent's most recent turn was right given everything said so far.
Turns 10 through 13 do not just stay bad, they get slightly worse each time, and the slope is worth a caveat. My reading is escalating commitment: each of those turns is graded on its own merits, and on its own merits each is a bigger claim than the one before it. Turn 10 asserts the refund, turn 11 quotes an amount, turn 12 promises a timeline, turn 13 reads out a confirmation number. Nothing is carried forward by the scorer; the agent is simply wrong about more, more specifically, each time it opens its mouth.
I should be honest that this is a reading of four points from one call and not a result. It could as easily have gone the other way: a rubric asking whether the latest turn was correct and appropriate might reasonably treat "here is your confirmation number" for a refund that does not exist as a second cliff rather than three points worse than the turn before, since inventing a confirmation number is a different severity class from repeating a wrong policy. I got the gentle ramp and I do not have a mechanism that predicts gentle over cliff. The test is sitting there in the other 40 calls, which should show a ramp where the agent escalates and a plateau where it just repeats itself, and I have not run it.
Then look at turn 14, which goes back up 14 points against turn 13. That is the closing turn, and my rubric scores a turn partly on whether it is well formed: acknowledges the customer, summarises, offers a next step. The agent did all three, on top of a wrong answer, and got paid for it. Some fraction of what my scorer measures is how gracefully the agent delivers bad information, and I would not have found that without the curve.
The fortnight I spent not trusting it
Cost first, because this is the objection I would raise.
Prefix scoring is O(n) calls to your scorer for an n-turn conversation, so a fourteen-turn call costs fourteen judge invocations instead of one. Across the 41 failures that was turn for turn about 470 extra judge calls. At the model we use for grading that was small money and roughly nine minutes of wall clock, run in parallel. On our full nightly suite it would not be small, which is why we do not run it there: prefix scoring is a debugging tool that runs on failures, not a gate that runs on everything. The gate still emits one number per call. When the gate goes red, the debugger goes and finds the turn.
You can also do it in log(n) instead of n if you bisect: score the first half, and if it is already bad recurse left, otherwise recurse right. I tried it. It found the same turn on 34 of the 41 calls and a different one on 7, and every one of the 7 was a call with two separate problems, where bisection commits to a side early and never sees the other one. Full scan for debugging, bisection if you are impatient and know your calls fail once.
Now the part that lies to you, and it took me a fortnight to see it.
A prefix is not a conversation. When you score turns 1 through 5 in isolation you are asking your rubric to grade a call that appears to end at turn 5, and most rubrics have opinions about endings. Mine did. "Did the agent resolve the issue" scores an unfinished conversation harshly for the simple reason that nothing has been resolved yet, so every early prefix carried a penalty that had nothing to do with quality. My first version of this curve sloped downward everywhere and I nearly threw the method out.
The fix was to grade prefixes against a rubric that asks a turn-local question instead of an outcome question. Not "was the issue resolved," which only makes sense at the end. Something closer to "given everything said so far, was the agent's last turn correct and appropriate." Same scorer, different prompt, and the curve went flat-then-cliff instead of monotonically down. The rubric you use for the gate is very likely the wrong rubric for the curve, and reusing it is what makes the method look broken.
Worth being explicit here, because I have argued something that sounds like the opposite. A few weeks ago I wrote about a seven-turn call where every turn graded in isolation was correct and the call still failed, and I used it to argue against turn-level grading. I still think that is right about grading turns in isolation, which is what that system did: it handed the judge one turn with no history. On turn four the agent confirmed a Tuesday to a caller who had said earlier in that same call that she could not do Tuesdays, and turn four read as a perfectly good confirmation to anything that could not see the turn where she said it. The rubric here is different. It grades the latest turn conditioned on the whole prefix, which is exactly the information the isolated version was throwing away, so it should have caught that one. I have not gone back and run it on that call, and I should. What I got wrong in July was blaming the granularity when the problem was the missing context.
Three more places it misleads. Turns where the agent says almost nothing ("sure, one moment") score noisily because there is very little to grade, and I now skip any agent turn under about five words rather than trust its number.
The min_drop threshold has a blind spot I should name, since it is the same shape as the bug that started all this. A call that degrades gradually, 0.84 to 0.71 to 0.58, has no single step reaching 0.15, so the function returns nothing at all and reports the largest drop it saw, 0.13, even though the call lost 26 points end to end. A slow slide is invisible to a detector that only looks one step at a time. Looking at the curve rather than the returned index catches it, which is an argument for plotting the thing rather than trusting the number that comes out of it.
And a conversation that fails because of something the agent never said, an omission rather than an error, does not produce a cliff at all. The curve just sits slightly low the whole way. I have not solved that one. Omissions remain the failure class I still find by reading.
What shipped, and what I'd tell past me
What shipped: prefix scoring as a debug command, run on demand against failed calls, with a turn-local rubric that is versioned separately from the gate rubric. The output is a turn index and a drop magnitude. It goes in the incident notes. Time from "this call failed" to "this turn, this cause" went from four to nine minutes of reading down to well under a minute.
I owe you a number on its reliability, because I spent a whole section above complaining that my own labels did not reproduce and it would be cheap to skip the same test on the tool. Temperature 0 does not buy you determinism here, incidentally. It makes the sampler greedy, which removes the sampling noise and nothing else. Two things still move a score between replays: floating-point reduction in the serving stack is not associative, so a change in how your request gets batched with other people's can shift the logits enough to flip an argmax at a near-tie, and the provider can move the model under a stable name. Both are outside your process. So it has to be measured rather than assumed.
I replayed all 41 calls three times at temperature 0. The identified turn was stable on 39 and moved on 2. Both of the unstable ones had their two largest candidate drops within about 0.04 of each other, so the detector was picking between near-ties rather than the judge being wildly inconsistent, and both of those calls show two visible steps on the curve rather than one cliff. That is a failure mode you can see, which is the property I actually wanted.
Second thing that shipped, and honestly the bigger win: when the mean score moves in production, we now re-run prefix scoring across a sample of the affected calls and look at the distribution of drop-turns. A regression concentrated at turn 2 and a regression spread evenly across turns 4 to 12 are different bugs with different suspects. I have not been able to go back and test that against the six-hour incident, because the affected calls aged out of our retention before I built any of this. It is the first thing I will run the next time the mean moves.
What I would tell past me: the granularity of your score is a design decision, and defaulting to one score per conversation is one of the choices, however little it feels like choosing. I spent a week reading transcripts because my tooling handed me a float and I assumed that was the shape the answer came in. It was just the shape my scorer happened to emit. The conversation was always a sequence and the failure was always at an index, and I could have asked for the index at any point in that week.
The other thing I would tell him is that the number going back up at turn fourteen was the tell. A score that improves at the end of a call that failed is measuring the shape of the answer as much as its content. I looked at that number for a week and read it as noise.
