Case Study: A Benchmark Gate Rejected a Free Model's C++ "Optimization" — and the Noise Floor Agreed

A free model proposed replacing std::unordered_map with a sorted vector. The benchmark gate rejected the patch. The candidate was 4.1% slower, not faster.

That is the outcome in one sentence. The rest of this case study explains why the gate existed, how it worked, and what the rejection taught me about accepting AI-generated C++ changes.

Background

I maintain a small C++17 tool that parses log lines into key-value buckets. Profiling showed one hot spot: a std::unordered_map<std::string, int> lookup consumed roughly 18% of total runtime.

The map held short keys, most under eight characters. The lookup ran in a tight loop over 200,000 records per batch.

I pasted the profile output into a free model endpoint available through MonkeyCode's free model access. The suggestion was reasonable on its face: replace the hash map with a sorted std::vector<std::pair<std::string, int>> and use std::lower_bound. Better cache locality, fewer allocations, simpler code.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Plausible reasoning is not evidence. I had been burned before by model patches that looked correct and regressed at runtime. So I built a gate.

Goal

The acceptance criteria were fixed before any code changed:

1. Median lookup latency must improve by at least 5%.
2. The improvement must be statistically significant (p < 0.05, Welch's t-test).
3. The result must hold across 15 interleaved A/B runs.

No subjective review. No "it feels faster." A number, or a rejection.

Implementation

The gate had three parts: a benchmark harness, a statistical comparator, and a CI step.

Step 1: Benchmark harness

A small bench.cpp compiled once per variant with identical flags:

// bench.cpp — compile once per variant, same flags
#include <chrono>
#include <iostream>
#include <string>
#include <vector>

extern int lookup(const std::string& key); // variant-specific

int main(int argc, char** argv) {
    const int rounds = argc > 1 ? std::stoi(argv[1]) : 20;
    std::vector<std::string> keys = load_keys("keys.txt"); // 200k entries

    auto start = std::chrono::steady_clock::now();
    volatile int sink = 0;
    for (int r = 0; r < rounds; ++r)
        for (const auto& k : keys)
            sink += lookup(k);
    auto end = std::chrono::steady_clock::now();

    double total_ns = std::chrono::duration<double, std::nano>(end - start).count();
    std::cout << total_ns / (rounds * keys.size()) << '\n';
    return sink == 0 ? 0 : 1;
}

load_keys is omitted for brevity; it reads the same 200,000 keys from a file for both variants.

Step 2: Interleaved runs

The key design decision was interleaving. Baseline and candidate ran alternately, pinned to the same core, so thermal drift and CI noise hit both sides equally:

for i in $(seq 1 15); do
  taskset -c 2 ./baseline_bench 20 >> baseline.txt
  taskset -c 2 ./candidate_bench 20 >> candidate.txt
done
python3 bench_gate.py baseline.txt candidate.txt --min-effect 0.05

The comparator is a small script. Simplified, it looks like this:

# bench_gate.py (simplified) — Welch's t-test + effect-size check
import math, sys

def stats(path):
    xs = [float(x) for x in open(path)]
    n = len(xs)
    m = sum(xs) / n
    var = sum((x - m) ** 2 for x in xs) / (n - 1)
    return m, var, n

b_m, b_v, b_n = stats(sys.argv[1])
c_m, c_v, c_n = stats(sys.argv[2])
se = math.sqrt(b_v / b_n + c_v / c_n)
t = (c_m - b_m) / se
delta = (c_m - b_m) / b_m
print(f"baseline={b_m:.2f} candidate={c_m:.2f} delta={delta:+.2%} t={t:.2f}")

The real version computes the p-value from the t-statistic and exits non-zero when the candidate fails the effect-size threshold. The printed verdict is the only signal the CI job needs.

Step 3: CI gate

The gate ran as a separate job, not inside the build job. It produced one line of output:

baseline: 41.2 ns/lookup (sd 0.8)
candidate: 42.9 ns/lookup (sd 0.9)
delta: +4.1%  p=0.003  verdict: REJECT

Results

The sorted vector was slower. Not by noise — by 4.1%, with p = 0.003.

Why? The keys were short, so hashing was cheap. The vector's binary search introduced branch mispredictions on every lookup. The model's cache-locality argument was theoretically sound and empirically wrong for this input distribution.

I fed the rejection output back to the model. Its second proposal was different: keep the hash map, add reserve() with the expected bucket count, and use a transparent hash for std::string_view lookups. The same gate measured a 7.2% improvement (p = 0.001), and the patch merged.

The free server option in MonkeyCode was enough for this loop. A disposable machine ran the 15-minute A/B sequence and shut down afterward. No persistent infrastructure, no shared CI runner noise.

Lessons learned

1. A gate turns opinions into falsifiable claims. The first patch was not wrong in any static-analysis sense. It was wrong for this workload, and only a measurement could say so.
2. Interleave or die. Running baseline first and candidate second, in separate jobs, measures thermal drift more than code. Interleaving is what made p = 0.003 meaningful.
3. Feed rejection data back. The second proposal was better because it saw the first one's numbers. Propose, measure, reject, re-propose — that loop is where free model access earns its keep.
4. Statistical significance is not practical significance. A 1% win with p = 0.0001 is still a 1% win. The 5% effect-size threshold blocked a merge that would have added complexity for nothing.

Limitations

This gate is narrow by design. It validates one function on one machine under one compiler. It says nothing about memory usage, code size, or maintainability.

Do not use this approach when:

• The hot path is I/O-bound. The benchmark will measure the disk, not your code.
• You lack a stable machine. Noisy shared runners produce p-values you cannot trust.
• The change is a one-liner. Setup cost exceeds the benefit.
• You need a decision in minutes. Fifteen interleaved runs take at least 15 minutes.

The gate exists for one specific case: a model proposes a performance change, and the team needs evidence before merging. That case is common enough, and cheap enough to automate, that I now run it for every AI-proposed optimization.

The first rejection was the best possible outcome. It proved the gate could say no.