What Happens When an AI System Is Built to Challenge Its Own Decisions?

This is a submission for the Kaggle Benchmarking Challenge

I built an AI benchmark where checking your own decision has a cost.

Then I tried three increasingly strong ways to make the system challenge itself:

1. tell it to seek counterevidence,
2. give it an executable way to test hypothetical consequences,
3. automatically put those consequences into its decision path.

None produced a reliable average improvement in the final decision.

That was not the result I expected.

And it became the most interesting part of the benchmark.

What I Benchmarked

Decision-Conditioned Falsification asks a model to act in a small world with three possible hidden mechanisms.

The model knows the possible mechanisms, but not which one is active.

It has a choice:

ACT now

or

TEST → pay a cost → observe evidence → decide

Sometimes testing is the right move.

Sometimes the exact same desire for more information is a waste of points.

That distinction is the benchmark.

It does not reward how much the model doubts itself, how many tests it runs, or how often it changes its mind.

It rewards:

net operational utility.

A simple way to think about it:

If checking costs 5 points, is knowing more actually worth 5 points before you act?

That also lets the benchmark separate three things that are often collapsed together:

• Information gain: how much did I learn?
• Falsification: did the evidence contradict my hypothesis?
• Decision value: did learning it actually help me make a better decision?

They are not the same thing.

Models Tested

The public Kaggle benchmark currently contains four model runs:

• GPT-5.4 nano
• Gemini 3.5 Flash
• Claude Haiku 4.5
• Gemini 3.7 Flash

Gemini 3.7 Flash was used during Kaggle task creation, so I preserved the run rather than silently removing it.

The other three were selected before the final evaluation to give the benchmark several model families instead of many variants from one provider.

The live leaderboard is available on Kaggle.

But the most interesting result came from using the environment to test the architecture around the model.

Findings

Here is the entire experimental story in one table:

That progression changed how I think about AI self-checking.

1. Asking for counterevidence was not enough

The first comparison held the model, environment, tools, budgets and output structure constant.

One condition simply added an explicit instruction to consider evidence that could contradict assumptions supporting its plan.

The average difference was:

−1.45 points

95% interval: [−2.93, +0.01]

So explicitly asking for falsification did not show a robust aggregate benefit.

The traces suggested another problem: the model sometimes declared predicted outcomes that were physically impossible under every mechanism in the environment.

Instead of making the prompt more forceful, I changed the system.

2. Making hypotheses executable was not enough either

I added a deterministic function that could answer:

If W1 were true, what would this experiment produce?

This was kept separate from actually observing the hidden world.

predict(hypothesis, experiment)
        ≠
run_experiment(experiment)

The first calculates a hypothetical consequence.

The second produces real evidence.

The surprise was simple:

the model almost never used the prediction capability.

It was invoked once across 360 assisted episodes.

That gave me a much clearer engineering lesson:

A capability being available inside an agent does not mean that capability participates in its decisions.

So I removed the choice to invoke it.

3. Even automatically supplying the consequences was not enough

In the final experiment, the model first proposed a decision.

Before that proposal could become an action, the system automatically calculated its consequences under the possible mechanisms and put them directly into the model's context.

The control received the same opportunity to reconsider, but without those computed consequences.

This time the model could not simply forget to open the tool.

The projection reached the model in 170 of 180 applicable first reviews.

The result:

Integrated consequences − Neutral review: −0.39 points

95% interval: [−1.53, +0.80]

At the first review itself, where both branches started from exactly the same provisional decision, the difference in decision value was just:

−0.088 points

95% interval: [−0.454, +0.402]

The intervals do not prove equivalence.

But I still did not observe an average improvement from inserting the correct hypothetical consequences into the decision path.

The result I actually care about

It is tempting to think of this as one chain:

The system can check something
            ↓
The check is correct
            ↓
The check reaches the model
            ↓
The model uses it
            ↓
The decision improves

My experiments made those steps come apart.

A check can be correct.

It can reach the model.

The model can reconsider.

And the final decision can still fail to improve.

That does not mean self-falsification is useless.

It means:

Correct epistemic machinery is not automatically valuable decision machinery.

A safeguard should not be considered successful simply because the safeguard itself works.

It has to improve what the system eventually does.

The harder problem was deciding when to check

Another thing surprised me.

The difficult part was often not finding a counterexample.

It was knowing whether another experiment was worth running at all.

Models sometimes stopped when another test still had positive decision value.

They also sometimes paid for information when acting immediately was already better.

An experiment can reveal a lot and still be useless for the current decision.

A hypothesis can be falsified without changing the best action.

And a model can obtain useful evidence and still choose the wrong action.

That gap now interests me more than simply adding another verification layer.

How does evidence become causally connected to action inside an AI-native system?

My Benchmark

Decision-Conditioned Falsification

👉 View the benchmark on Kaggle

The benchmark is published under Apache 2.0.

One display note: Kaggle currently renders the raw task value in the leaderboard table using percentage-style scaling, while the benchmark summary displays the underlying value correctly. I kept the frozen task unchanged and use the underlying score in the benchmark documentation.

I started with this question:

Can an AI system prove its own decision wrong when checking has a cost?

After building the benchmark, I would ask a different one:

Can we design the path from hypothesis, to evidence, to action so that discovering you are wrong actually improves what you do next?

That became the more interesting benchmark.