A red Playwright build can't tell you whose fault it is

Every failing end-to-end run asks the same question, and the report never answers it:

Did the app break, or did the test go stale?

It matters more than anything else on the screen. One answer means stop the merge and fix a regression. The other means the app changed on purpose and the test is describing a version of the product that no longer exists. Same red X. Opposite response.

Here's what Playwright hands you instead:

Error: locator('[data-testid="place-order"]') timed out after 5000ms

That's a fact about a selector. It isn't a fact about your product. And you're the one who has to close the gap — open the trace, remember what that button does, go find the PR, read the diff, decide.

I got tired of doing that by hand, so I tried to make CI do it. This is what I learned.

The stack trace is missing the one thing you need

A Playwright failure knows the selector, the timeout, and the line number. It doesn't know what place-order is. It has no idea that button is the last step of a checkout flow, or that this PR deliberately hid it behind a new express-checkout summary.

The information isn't hidden or hard to get. It just never gets written down anywhere a machine can read it.

test.step() doesn't fix this, and it's worth being precise about why

The usual answer is to wrap things in test.step('add two items to the cart') and get a nicer report. It does help — the report reads better.

But that string has two problems.

You wrote it by hand. So it's another thing to maintain, and the moment someone changes the code without changing the string, your report is confidently describing behaviour that no longer exists. A stale description is worse than none, because you trust it.

It's prose. 'add two items to the cart' is readable, but there is nothing in it a program can match against a diff. It doesn't name a control. It doesn't carry a selector. You can't join it to anything.

So you get a nicer report and you're still the one answering the question.

Name the control once

The change that made everything else possible is small: instead of describing the step, name the control.

const email      = new Input(page, 'email', 'Email');
const cartItems  = new Table(page, 'items', 'Cart items');
const placeOrder = new Button(page, 'place-order', 'Place order');

await email.fill('ada@example.com');
await cartItems.expectRowCount(2);
await placeOrder.click();

Each control knows three things: what kind of thing it is, what selector it resolves to, and what a human calls it. Nobody writes a sentence. Nobody maintains a sentence.

From that, the sentence falls out for free:

✓ Type "ada@example.com" into field "Email"
✓ Assert table "Cart items" has 2 rows
✗ Click button "Place order"

That's the part people notice first, and it's the least interesting part. The readable report is a side effect. The real output is this:

{
  "action": "click",
  "controlType": "button",
  "name": "Place order",
  "selector": "[data-testid=\"place-order\"]",
  "status": "failed"
}

Now the run has left behind a machine-readable record of which parts of the UI the test actually touched at runtime — not which lines of code executed. That distinction is the whole thing.

The join

Once a failing run records controls by selector, and a pull request is a set of changes to markup and test IDs, you have two sets with a shared key.

THE TEST TOUCHED          button "Place order"  →  [data-testid="place-order"]
THIS PR CHANGED           the element behind [data-testid="place-order"]

Intersection non-empty → the PR changed the exact thing this test was reaching for. The test is describing the old world. Stale test.

Intersection empty → the test failed on something this PR never touched. Regression candidate.

That's it. That's the mechanism. It's almost embarrassingly simple, and it's only available because the control was named before the run started.

The mistake that taught me the most

The first version of this shipped the failure straight to an LLM: here's the error, here's the trace, tell me what happened.

I ran it against a demo PR that hid the Place order button on purpose and removed a cart row. Two tests failed. The analysis came back several paragraphs long, extremely well written, and said the failures came from a bug in a shared beforeEach where a second add-to-cart was silently not awaited, leaving the cart half-populated.

There is no such bug. The fixture is fine. The PR removed the row deliberately — it says so in its own description.

The model wasn't hallucinating in the usual sense. It was doing exactly what I asked: reasoning about a failure from the failure alone. Given only the symptom, a broken fixture is a genuinely reasonable hypothesis. It was reasoning well from insufficient evidence, and there is no prompt that fixes that.

What made it dangerous was the confidence. A vague answer wastes ten seconds. That answer sends someone hunting through a healthy fixture for an hour, and when they find nothing, they stop trusting the tool.

So I moved the evidence. Before any model runs, plain code takes the controls each failing test touched — name and selector, straight out of razo-steps.json — and looks for them in the added and removed lines of the diff. String containment, no inference. That intersection is what the PR comment now shows as evidence, which is why you can check it yourself: anyone can confirm the diff touched place-order.

The model still writes the verdict. But it's no longer reasoning from a symptom — it's reasoning from the change that caused it, with the hypothesis space already collapsed. A separate check vetoes any category it invents outside the allowed vocabulary.

I'll be honest about where this stops. The join is deterministic; the classification is still probabilistic. Nothing yet verifies that the category the model picked is consistent with the evidence the code assembled — that rule lives in the prompt, which is an instruction, not a mechanism. It behaves correctly today, including against a PR with an empty description, but that's an observation about behaviour rather than a structural guarantee. The next step is to close it: if the model returns intentional change while the intersection is empty, code should overrule it, because the model is contradicting evidence that was computed, not inferred.

Which leaves an uncomfortable asymmetry worth stating plainly: the part nobody else has is the deterministic one, and the part that decides what the user is told is the probabilistic one. That's the wrong way round, and it's the thing I'm fixing next.

What this still can't do

Honest limits, because tools that only list strengths are advertising:

It needs the controls named. No named control, no artifact, no join, no verdict. That's a real adoption cost — you're editing a suite that already works. The mitigation is that it's incremental: name the controls in the flow that breaks most often, leave everything else alone. Plain Playwright calls keep working, they just don't narrate.

It only reasons about what the test touched. If a PR breaks something the test reaches indirectly — a shared component three levels up, a changed API response — the selector-level join won't see it. That's the ambiguous bucket, and it's where the model earns its place.

No PR, no verdict. A nightly or a local run has no diff to read against. You still get the narrated story of what the test did, which is useful, but the question at the top of this post needs two inputs and one of them is missing.

None of this is clever. It's one decision — write down what the test touched, in a form a program can read — and then everything downstream stops requiring a human. The readable report was the accident. The join was the point.

The framework is MIT on GitHub if you want to read the implementation: github.com/razohq/razo