An agent evaluation fails in CI but passes locally. Before blaming the model, ask whether both runs saw the same tool responses, database state, clock, configuration, and dependency versions.
Containers cannot make an external model deterministic. They can remove a large amount of accidental variability around it.
I use Docker Compose as an evaluation lab: a small, versioned environment that can replay synthetic cases, provide controlled tools, inject faults, and preserve evidence.
Separate the core from optional experiments
Compose profiles are useful when the same lab supports several modes. Services without a profile start normally; profiled services start only when enabled or explicitly targeted.
services:
fixture-api:
image: ghcr.io/example/agent-fixtures@sha256:REPLACE_WITH_DIGEST
environment:
FIXED_NOW: "2026-09-01T12:00:00Z"
DATASET_PATH: /fixtures/cases.json
volumes:
- ./fixtures:/fixtures:ro
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"]
interval: 2s
timeout: 1s
retries: 20
eval-runner:
build:
context: .
dockerfile: Dockerfile.eval
depends_on:
fixture-api:
condition: service_healthy
environment:
FIXTURE_URL: http://fixture-api:8080
EVAL_SEED: "1701"
volumes:
- ./fixtures:/app/fixtures:ro
- ./evidence:/app/evidence
profiles: [eval]
fault-proxy:
image: ghcr.io/shopify/toxiproxy@sha256:REPLACE_WITH_DIGEST
profiles: [fault]
otel-collector:
image: otel/opentelemetry-collector@sha256:REPLACE_WITH_DIGEST
profiles: [observe]
The digests are placeholders: resolve and commit real image digests in your repository. Keep core dependencies such as the fixture service unprofiled. Make optional runners, fault injectors, local models, or telemetry services explicit.
Run the basic lab:
docker compose run --rm eval-runner
Add an optional fault service when testing timeouts and retries:
docker compose --profile fault run --rm eval-runner
Compose automatically starts a targeted profiled service and its declared dependencies. Multiple --profile flags can combine modes.
Version more than the application image
A replayable case needs an environment manifest:
{
"caseId": "refund-approval-required",
"datasetRevision": "fixtures-2026-09-01",
"promptRevision": "refund-v7",
"policyRevision": "policy-v3",
"toolSchemaRevision": "tools-v5",
"model": "configured-model-id",
"temperature": 0,
"seed": 1701,
"clock": "2026-09-01T12:00:00Z"
}
Record the values even when a provider does not guarantee seed determinism. The manifest describes the attempted conditions; it does not promise identical tokens.
Pin these inputs where practical:
- container images by digest;
- JavaScript dependencies with a lockfile and frozen install;
- synthetic datasets by content hash;
- prompt, policy, and tool-schema revisions;
- locale, timezone, clock, and random seed;
- provider and model identifiers.
Make tools reproducible before judging the agent
Suppose lookup_order normally calls a changing production API. In the lab, route it to fixture-api and select behavior by case ID:
const result = await fetch(
`${process.env.FIXTURE_URL}/orders/synthetic-42`,
{ headers: { "x-eval-case": "refund-approval-required" } },
);
The fixture should return the same body, status, and artificial delay for that revision. A fault profile can then introduce a deliberate timeout or malformed response.
This gives failures meaning. If the agent skips authorization against a fixed fixture, the behavioral regression is easier to distinguish from a backend change.
Persist evidence outside the containers
Do not let the only result be a process exit code. Write a bounded artifact set to the mounted ./evidence directory:
evidence/
environment.json
results.json
junit.xml
traces/
README.md
The README should identify the command, fixture revision, expected invariant, and any redaction performed. CI can upload the directory even when the runner fails.
Add a machine-readable comparison file when a release candidate is evaluated against a baseline:
{
"baseline": "prompt-v6",
"candidate": "prompt-v7",
"dataset": "fixtures-2026-09-01",
"casesAdded": 0,
"casesRegressed": ["refund-approval-required"],
"casesImproved": ["policy-timeout-recovery"],
"unknown": []
}
Do not collapse that into one average score. A candidate can improve ten harmless cases and regress one protected action.
Make the evidence directory unique per run and write it atomically. Reusing ./evidence/results.json across parallel CI jobs can combine a new result with an old environment manifest. A simple run identifier and completion marker make partial output obvious:
evidence/run-2026-09-09-1701/
environment.json
results.json
comparison.json
COMPLETE
Create COMPLETE only after every required artifact is flushed. Upload incomplete directories for diagnosis, but never grade them as finished evaluations.
If you use a local trace tool such as AgentInspect, keep it optional and metadata-first. The lab architecture should not depend on a hosted viewer, and captured prompts or tool payloads still require a sharing review.
Know what Compose does not solve
The lab does not reproduce provider load, a production vector index, browser scheduling, or every distributed race. A temperature of zero is not a mathematical guarantee of identical output. Container isolation is also not a security assessment of the code being executed.
Its value is narrower: it makes controlled inputs explicit and repeatable enough to evaluate stable outcomes and trajectories.
Targeting eval-runner explicitly enables its Compose profile and starts declared dependencies, but it does not enable unrelated profiled services. Keep profile combinations in named scripts or CI jobs so a developer does not accidentally compare a fault-injected run with a clean baseline.
When an agent test changes, you want the first question to be “Which behavior changed?”—not “Was my laptop using a different clock, dataset, or tool server?” A small Compose lab moves many of those variables into versioned engineering decisions.
