The Agent Said It Worked. I Asked the Kernel.

Before evaluating an agent’s code, I built a backup client with eight known behaviors to check the instrument itself.

My response to “it works” is becoming: “Let me see the packet capture.”

This may become a personality problem. For now, it is an experiment.

This experiment was inspired by Hemapriya Kanagala (@hemapriya_kanagala) and her article, What Happens When AI Outgrows the Tests We Use to Measure It?. Her question—“92% according to what?”—stayed with me. She examines how evaluation can become less informative as benchmarks saturate, reference answers become complicated, and test conditions change.

I wanted to take one piece of that discussion to the workbench: what independent evidence supports a program’s claim that it succeeded?

So, with AI assistance, I built a small native backup client, gave it eight deliberately chosen behaviors, and observed it from outside its own logging. Some variants produce the right file while doing questionable things along the way. One cheerfully reports success without backing anything up.

The kernel has very little appreciation for cheerful reporting.

The claim I had to correct

My first instinct was to declare that, until someone redefines how computers work, we have two sources of truth: the CPU and the network.

It sounded excellent in my head. Then the engineering questions arrived.

A CPU can faithfully execute the wrong algorithm. A packet capture can faithfully record the wrong bytes reaching the wrong destination. The saved file matters too. And none of those observations knows what the user actually requested.

The more defensible position is that execution, network activity, and resulting state provide evidence we can judge against a requirement. They have different coverage and different blind spots.

For this experiment, the requirement begins with something wonderfully unromantic: the saved backup must match the source file.

Now we have something to measure.

Cheap code makes verification more interesting

Frameworks and libraries have helped us build software without personally supervising every system call. That remains a useful division of labor.

But familiar frameworks do not automatically validate unfamiliar code assembled on top of them.

An agent can help produce an implementation, tests, and an explanation of the passing tests. If the implementation and its tests share a misunderstanding, agreement between them can be misleading.

Humans can do this too. We have been writing tests that flatter our implementations since before the current generation of autocomplete had electricity.

My concern is the growing volume of plausible implementations we can produce and the time available to examine them. Independently collected evidence gives us another way to challenge the result.

Sometimes that evidence is simply a file comparison. Sometimes the output is correct and we need to investigate how the program got there.

A backup client small enough to understand

The fixture is intentionally modest:

• A native C client reads a deterministic 512 KiB file.
• A Python receiver listens on loopback TCP.
• The client computes a SHA-256 digest: a fingerprint used to check content equality.
• Before uploading, it asks whether the receiver already has matching content.
• The receiver validates incoming content before committing it.
• A separate verifier computes the saved file’s digest and compares it with the source.

Each scenario runs the client twice against the same receiver state. The baseline uploads on the first invocation and recognizes unchanged content on the second.

The contract we want the baseline to satisfy is explicit: save matching content, send no upload payload on an unchanged second invocation, contact only the configured receiver, and recover from the injected first-transfer interruption within three upload attempts. An upload whose payload does not match its advertised digest must be rejected before it replaces the saved backup.

Repeated hashing and spinning are diagnostic cases for unnecessary work. We have not set a performance threshold or measured a speedup here.

A second local endpoint lets us demonstrate an unexpected connection without contacting an outside service.

The eight behaviors are deliberately seeded demonstrations. They check whether our observers detect known behaviors. They do not measure how often an AI agent introduces these defects, and they are not evidence about a particular model’s coding ability.

What the observers can see

The utility collects several kinds of evidence. Their distinctions matter more than the sophistication of their names.

System calls: what the client asks the kernel to do

strace records system calls: operations through which a process requests services such as reading files or opening connections. In this lab, it records timestamps, call durations, decoded file descriptors, and return values.

That lets us count bytes returned by reads of the source file. These are logical reads, not physical disk traffic; the operating system may satisfy them from memory.

eBPF: selected kernel events and function entries

eBPF supports observation at hooks such as system calls, kernel tracepoints, and function entry or exit. The lab uses bpftrace to count selected events for the native client. eBPF’s introduction explains the underlying mechanism.

A userspace probe, or uprobe, observes a point in an executable. Here is the probe that counts entries into our hashing function:

uprobe:__BINARY__:hash_file /pid == cpid/ {
    printf("%llu pid=%d hash_file\n", nsecs, pid);
    @hash_calls = count();
}

The runner replaces __BINARY__ with the executable’s path. cpid identifies the child launched through bpftrace -c, so the filter restricts these observations to that client. bpftrace documents this built-in here.

Other probes observe connection calls, successful read/send byte counts, scheduling events, and entries into the payload-send and busy-wait functions.

An entry counter tells us that control reached a function. It does not establish that the function returned the correct answer. That is why the outcome check remains separate.

CPU samples: where execution spends its work

The host used for this demonstration has an AMD Threadripper PRO 5975WX. Linux exposed AMD Instruction-Based Sampling through ibs_op, and the utility successfully collected it through perf.

The saved artifacts include CPU call-stack profiles and assembly annotated with sample weights. These let us inspect execution inside the client, its libraries, and sampled kernel paths.

This is sampling. It is not a recording of every instruction, every register, or every intermediate value.

Packets: traffic at the capture point

tcpdump captures loopback traffic filtered to the two fixture ports. We preserve the packet file, a readable packet summary, and the capture tool’s drop counters.

Receiver payload counts are a different measurement: bytes the application consumed. A successful send can hand data to the local kernel before the receiver consumes it. Headers, retries, and buffering also affect what each observer sees.

Keep those quantities separate. Otherwise, the instrument starts manufacturing the confusion it was built to investigate.

Three ways a green checkmark can be incomplete

1. Success without a backup

The false-success variant prints a completion event and exits with status zero. It does not upload the file.

The independent verifier finds no saved backup. In the eBPF run, there are no observed client connection calls, and the filtered packet capture contains zero packets.

This is the bluntest example in the collection. A test that checks only the exit status would accept it. A test that verifies the resulting file would reject it immediately.

We did not need a CPU probe to discover that the file was missing. The low-level observations help establish what accompanied that failure. Use the simplest independent check that answers the question.

2. A correct backup, with 256 helpings of hashing

The hash-storm variant saves a backup whose digest matches the source. It also hashes the entire source 256 times per invocation.

On the second invocation, the system-call trace records 134,217,728 logical source bytes read for a 524,288-byte file:

134,217,728 / 524,288 = 256

The receiver consumes no upload payload on that invocation. The content is unchanged; the unnecessary work happens before that decision.

A separate eBPF run observes 256 entries into hash_file on each invocation. Here are the final counters, copied verbatim from its second invocation’s raw probe output:

@connects: 1
@hash_calls: 256
@read_bytes: 134233569
@sent_bytes: 75
@switches_out: 4

The read counter is slightly larger than the source-file total above: this probe counts successful read bytes across the client’s descriptors, while the system-call summary selects reads of the source file. The send counter includes the digest-check request; zero upload payload does not mean zero network activity.

A separate AMD IBS run collects 233 CPU samples on its first invocation, reports zero lost samples, and places most inclusive sample weight beneath hash_file in the call-stack profile. “Inclusive” includes work in functions called by hash_file, such as the hashing library and file-read paths.

The samples help locate the work. The syscall and function-entry counts establish the repetition. This run does not quantify how much faster a corrected implementation would be.

These are observations from separate runs of the same seeded behavior. They corroborate the explanation; they are not one combined trace.

The output is correct. The computer has simply been asked to check the same pocket for its keys 256 times.

3. A correct backup, with an additional destination

The unexpected-egress variant makes a harmless connection to our second local endpoint before performing the backup.

The file still verifies correctly.

In the eBPF showcase, baseline connection counts are two on the first invocation and one on the second: a digest check plus an upload, followed by a digest check alone. The extra-connection variant records three and two.

Counts tell us there is more connection activity. The socket trace, packet addresses, and second endpoint’s records establish where it goes.

That distinction matters. Three connections do not inherently mean something is wrong. A destination requirement gives the observation its meaning.

The complete behavior set

Here is the compact view of the captured demonstrations:

In the retry scenario, the receiver disconnects after consuming 65,536 payload bytes. The client restarts from byte zero, and the receiver eventually commits the complete 524,288-byte file. Its total consumed payload for that invocation is 589,824 bytes.

That demonstrates bounded retry recovery. It does not demonstrate partial-transfer resume, which this implementation does not provide.

Run the experiment

The original measured source and captures are preserved alongside the polished implementation. The numbers above belong to that historical build. New runs may have different instruction addresses, timings, and sample counts.

Get the source and evidence from this pinned publication revision:

git clone https://github.com/copyleftdev/execution-evidence-lab.git observability-lab
cd observability-lab
git checkout --detach e61aadee4a68e6a1d5b54a00387dcd325b5d5e51

Then build the client and run the tests:

make
make test
python3 -m instrument run --all

The core dependencies are Linux, Python 3.10 or newer, a C compiler, Make, and the OpenSSL development library. Install the optional tracing tools for the collectors you want to use.

The runner prints a path to a Markdown report. Its default process collector is strace when available, and packet capture is best effort. Missing capabilities are recorded explicitly.

To select a collector:

python3 -m instrument doctor
python3 -m instrument run hash-storm --collector strace

On a trusted local lab machine, the privileged demonstrations can be run with:

sudo python3 -m instrument run --all --collector bpftrace --packets required
sudo python3 -m instrument run hash-storm --collector ibs --packets required

These commands run the synthetic lab as root. The fixtures use loopback and nonsecret generated data; the protocol is plaintext. The utility does not change host tracing policies. Hardware sampling and eBPF availability depend on the machine and its permissions.

For a run without tracing:

python3 -m instrument run --all --collector none --packets off

A successful showcase command means the scenarios executed, including the deliberately failing ones. Inspect each scenario’s independent verification result; do not interpret the runner’s exit status as “every backup was correct.”

The instrument needs scrutiny too

During development, the CPU-report parser initially matched the Samples portion of Total Lost Samples and displayed zero samples even though the raw profile contained hundreds.

The hardware data was present. Our summary was wrong.

Inspecting the raw artifact exposed the mistake. The parser was corrected, and regression tests now distinguish actual sample counts from lost samples. At the time of these captures, the 11-test suite checked fixture behavior and CPU-summary parsing; the saved traced runs provided separate collector validation. The public-release version adds protocol and collector-failure regression tests.

An article about distrusting convenient summaries was nearly defeated by its own convenient summary. There is probably a Unix utility for that feeling.

Other limits are less entertaining:

• Tracing changes execution. The recorded process CPU accounting includes collector overhead. These showcase runs are not controlled performance benchmarks.
• CPU samples can miss short-lived activity. An unsampled function may still have executed.
• The eight eBPF-showcase packet captures reported zero kernel drops. That is useful loss accounting, not proof of universal observation coverage.
• The normalized timeline contains client claims, receiver events, and verifier checks. Kernel and CPU traces remain separate artifacts. Different clocks require careful alignment.
• The verifier checks the receiver’s saved file. It does not exercise a separate restore command or prove survival through power loss.
• The observers are separate from the client’s reporting, but they share a host and were built as part of the same project. This is not an independent third-party audit.

The fixture also omits TLS, compression, multi-file snapshots, and source mutation during transfer. It observes a local program; it cannot see computation inside a remote model provider.

Those boundaries define what the results can support.

What this changes about evaluating an agent

The next experiment is to give an agent a clean implementation and ask it to make repeated backups faster while preserving integrity, recovery behavior, and destination restrictions.

Before that run, freeze the requirements, fixtures, resource limits, and grading criteria. Keep the independent checks outside the agent’s editable workspace. Preserve its patch, the executable identity, and the evaluation setup.

Measure performance with repeated untraced runs. Use traced runs to investigate differences. The seeded cases give us known behaviors against which to check the instrument first.

That is the connection back to Hemapriya’s article: the measurement needs to remain connected to the work we actually care about. For this small backup task, a correct file is necessary. Recovery, resource use, and destination behavior tell us more about the implementation’s suitability.

We can build those properties into better tests. Execution evidence helps us discover which properties our current tests leave out and investigate why a result occurred.

Cheap code makes it easier to produce a plausible solution. The engineering work includes deciding what evidence would make us trust it.

The agent can say it worked.

I would still like to see the file.

Acknowledgment

Thank you to Hemapriya Kanagala for the article that prompted this experiment. The instrumentation approach and its conclusions are my response; they should not be read as claims she made or an endorsement by her.

Hemapriya KanagalaFollow

Hey, I'm Hema 👋 Developer, writer, and creator of Dev Opportunity Radar, a weekly series published every Friday on DEV, helping people discover opportunities they might otherwise miss.

How this was made

AI assisted with the implementation, experiment execution, and drafting of this article. The cover illustration was AI-generated. The numerical observations above come from saved local runs; the behaviors were deliberately seeded. This is a demonstration of an evaluation instrument, not a model benchmark.