From Validating Code to Evaluating Behavior: Rethinking Testing for AI Agents

Traditional software testing assumes that system behavior is governed by deterministic code paths and configurations. Once an LLM enters the execution loop, that assumption weakens. The challenge shifts from verifying that code followed the expected path to evaluating whether a system behaved appropriately and achieved the desired outcome.

That distinction fundamentally changes what we test, how we write assertions, and even what "correctness" means. We are moving from verifying deterministic code to evaluating probabilistic system behavior.

Traditional Code Testing and Where It Starts Breaking Down

In traditional software systems, behavior is encapsulated into methods, APIs, configuration, and external dependencies. Even when a system becomes extremely complex, the behavior is still ultimately determined by code.

The output of a request flowing through a service can be largely thought of as a combination of:

Output ≈ Input × Configuration × Infrastructure × Dependencies × Code Version

The resulting state space may be enormous, but it remains largely open to systematic testing against an enumerated set of cases. This is the foundation upon which software testing is built. Cover enough of the state space and you gain confidence that the system will behave correctly.

A Concrete Example

Consider a workflow that reads configuration from a config store, calls a customer profile API, calls a pricing API, applies deterministic business rules, and produces a recommendation. Testing this system is relatively straightforward. We can:

• Mock the APIs
• Control configuration state
• Enumerate edge cases
• Verify outputs

Distributed systems introduce their own challenges—network latency, node failures, and partial outages—but established testing techniques such as chaos engineering can provide confidence that the system behaves correctly under those conditions.

Where Things Change with LLMs

Now replace the deterministic business rules with an LLM-driven planner. The APIs, tools, and data remain unchanged, but the workflow must now decide:

• Which tools to call
• In what order
• Whether more information is needed
• How to handle ambiguity
• How to formulate the final response

The upside is obvious: the system becomes dramatically more flexible and can handle scenarios that were never explicitly programmed for.

The downside is that the testing surface area grows significantly. The behavior driver is no longer only code; it is now a probabilistic reasoning system operating inside the application itself. Even temperature-0 inference does not always guarantee perfectly reproducible outputs in many production LLM deployments due to model updates, infrastructure differences, and implementation details.

The result is a system whose behavior is harder to predict, harder to bound, and therefore harder to test.

The Testing Challenge Shifts

From:

"Did the code execute exactly as expected?"

To:

"Did the system behave appropriately?"

Once we accept that agent behavior cannot always be validated through deterministic assertions, the next question becomes: what exactly are we evaluating?

What Are We Actually Testing?

Agent testing significantly expands what correctness means. Instead of asking whether a function returned the correct value, we ask whether the overall behavior was acceptable. A response can be factually correct while being incomplete, violate instructions, or arrive at the right answer through an inefficient or risky sequence of actions.

As a result, evaluating an agent often means evaluating it across multiple dimensions:

Outcome Metrics

These metrics focus on whether the agent successfully completed the task.

• Response Completeness — Did the agent fully complete the task, or only address part of the request?
• Response Correctness — Is the answer factually accurate, grounded in the available context, and compliant with instructions?

Operational Metrics

These metrics evaluate how efficiently the agent performed.

• Runtime and Latency — How quickly does the agent respond and complete its work?
• Cost Efficiency — How many tokens, model calls, and external resources were required to complete the task?

Reliability Metrics

These metrics measure how the system behaves when things do not go according to plan.

• Failure Handling — Does the agent recover gracefully from tool failures, timeouts, malformed inputs, and missing data?

Quality Metrics

These metrics focus on the usefulness and presentation of the final output.

• Response Quality — Is the output clear, concise, well-structured, and appropriate for the audience?

Governance Metrics

These metrics ensure the system operates within acceptable boundaries.

• Security and Safety — Is the agent resilient to prompt injection, jailbreaks, data exfiltration attempts, and other adversarial inputs?

Behavioral Metrics

These metrics evaluate how the agent arrived at its answer, not just the answer itself.

• Trajectory Quality — Did the agent follow a sensible path to reach its answer, or waste effort on unnecessary reasoning and tool calls?

The Paradigm Shift

Taken together, these dimensions reveal an important shift:

We are no longer evaluating whether a piece of code returned the expected value. We are evaluating whether an intelligent system behaved effectively across a range of often competing objectives.

How Do We Measure These Metrics?

Once we know what we want to evaluate, the next challenge is figuring out how to measure it.

In traditional software testing, assertions are usually deterministic. We know the expected output ahead of time and can compare actual behavior against it.

assert calculate_tax(100) == 18

Many agent evaluation dimensions do not naturally translate to this style of "hard" testing.

How do we write deterministic assertions for questions such as:

• Was the answer complete?
• Was the response faithful to the provided context?
• Did the agent use retrieved information appropriately?
• Was the tone professional?

These are semantic questions rather than computational ones.

As a result, modern AI evaluation frameworks have largely converged on three broad approaches.

1. Mathematical Metrics

Some dimensions can be measured using statistical techniques.

For example, here's an excerpt from the RAGAS metrics documentation on Answer Relevancy:

The Answer Relevancy metric measures how relevant a response is to the user input. It ranges from 0 to 1, with higher scores indicating better alignment with the user input.

An answer is considered relevant if it directly and appropriately addresses the original question. This metric focuses on how well the answer matches the intent of the question, without evaluating factual accuracy. It penalizes answers that are incomplete or include unnecessary details.

This metric is calculated using the user_input and the response as follows:

1. Generate a set of artificial questions (default is 3) based on the response. These questions are designed to reflect the content of the response.
2. Compute the cosine similarity between the embedding of the user input (E_u) and the embedding of each generated question (E_q).
3. Take the average of these cosine similarity scores to get the Answer Relevancy.

Answer Relevancy = (1/N) Σ cosine_similarity(E_q, E_u)

You can find the entire catalog of metrics provided by RAGAS here

2. Specialised Models

Some evaluation dimensions are measured using dedicated machine learning models trained for specific tasks.

Examples include:

• Sentiment analysis
• PII detection
• Bias detection

In these cases, the evaluator itself is a model that has been optimised for a particular classification or scoring task.

3. LLM-as-a-Judge with Pre-defined Rubrics

One of the increasingly common approaches is using another LLM to evaluate the output of the system under test.

Rather than comparing outputs against exact expected values, an evaluator model is asked to assess whether a response satisfies a particular rubric.

Frameworks such as DeepEval and RAGAS package these techniques into reusable metrics that engineers can apply directly within their test suites.

For example, rather than implementing faithfulness scoring from scratch, a framework may expose a pre-built metric:

from deepeval.metrics import FaithfulnessMetric

metric = FaithfulnessMetric()

The result is that engineers can focus on defining what good behavior looks like for their application rather than building evaluation logic from first principles.

Building Metrics on Top of These Approaches

This is an important shift. In traditional software, we primarily wrote assertions against outputs. In agent systems, we increasingly write assertions against behaviours, and those behaviours are often quantified through metrics rather than exact expected values.

Traditional Testing Layers Still Matter

Most traditional testing techniques remain just as important. The difference is that they now validate different parts of the system. An agent is still built on top of code, APIs, databases, configuration, authentication systems, and external dependencies. These deterministic components should continue to be tested using the same techniques that have worked for decades.

• Unit tests validate building blocks like API clients, parsers, and retry logic.
• Integration tests verify tool contracts and connections to databases or vector stores.
• Functional/E2E tests validate outcomes, ensuring the agent achieves goal Z given input X and environment Y.
• Chaos and Performance tests ensure the system remains performant and recovers gracefully from tool outages or timeouts.

Traditional testing therefore does not disappear in agent systems. It continues to provide confidence in the deterministic layers of the stack.

Testing Methodologies for AI Agents

Understanding what to measure is only half the problem. The next challenge is designing tests that produce meaningful and repeatable signals.

Define the Test Unit

The first step is deciding what a test actually represents. For agents, a test is usually centered around a scenario or behavior.

A typical test contains:

1. A scenario definition describing the behavior being evaluated
2. An input prompt that triggers the scenario
3. A controlled environment consisting of tool responses, configuration, and datasets
4. A set of assertions used to evaluate the outcome

Control the Environment

One of the easiest ways to create flaky agent tests is to allow the environment to change between runs.

Tool responses, configuration state, datasets, and external dependencies should be controlled wherever possible. This is no different from traditional testing, but becomes even more important because the LLM itself already introduces variability.

Without environmental control, it becomes difficult to determine whether a failure was caused by the agent or by a changing dependency.

Hard vs Soft Assertions

Some behaviours are non-negotiable and should fail immediately when violated. Examples include:

• Invalid schemas
• Incorrect computed values
• Required tool usage
• Missing mandatory fields

These are hard assertions.

Other behaviours are inherently subjective and are often better represented as scores or thresholds. Examples include:

• Completeness
• Clarity
• Tone
• Faithfulness
• Trajectory quality

These are soft assertions.

In practice, most agent tests combine both.

Account for Variability

Because agent behavior is probabilistic, a single test execution is rarely sufficient. Instead of asking:

"Did the test pass?"

We must ask:

"How consistently does the agent pass?"

Reliability is measured through success rates across multiple runs rather than binary outcomes.

Common Capabilities Across Agent Testing Frameworks

After exploring several agent evaluation frameworks, I found that most of them converge on a similar set of capabilities.

Whether you're looking at DeepEval, RAGAS, LangSmith, Arize or other emerging tools, the implementation details vary but the core ideas remain largely the same.

Most frameworks focus on three major areas:

1. Capturing and understanding agent behavior
2. Evaluating that behavior using reusable metrics
3. Persisting and re-running evaluations over time

Tracing and Observability

Most modern frameworks provide some form of tracing or instrumentation. This enables understanding:

• The sequence of tool calls
• Retrieved documents
• Intermediate decisions
• Model interactions

These captured data points provide the raw signals from which many evaluation metrics are derived.

For example, DeepEval allows developers to instrument portions of an application and collect execution traces that can later be inspected or evaluated:

from deepeval.tracing import observe

@observe
def retrieve_documents(query):
    ...

Metrics and Assertions

Once behavior can be observed, the next challenge is evaluating it.

Most frameworks provide a collection of reusable metrics that abstract away the complexity of evaluation. Instead of building custom scoring logic for every use case, teams can compose existing metrics into test assertions.

A typical evaluation for DeepEval, for example, may look like:

from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

metric = FaithfulnessMetric()
test_case = LLMTestCase(
    input="What is Kubernetes?",
    actual_output=response,
    retrieval_context=context
)
metric.measure(test_case)
assert metric.score > 0.8

The important idea here is not the specific metric itself, but the abstraction. Engineers are no longer writing assertions directly against outputs. They are increasingly writing assertions against behavioural qualities.

Test Datasets and Regression Testing

The final pillar is persistence.

Once tests have been defined, they need to be stored, versioned, and re-executed as systems evolve.

Prompt changes, model upgrades, workflow modifications, tool additions, and retrieval improvements can all introduce unexpected regressions. Without a persistent evaluation suite, it becomes difficult to determine whether changes are improving the system or silently degrading it.

A test case becomes a reusable artifact containing:

• Inputs
• Context
• Expected behavior
• Evaluation criteria

Once persisted, these datasets become regression suites that can be executed repeatedly throughout development and deployment.

Observability and Self-Learning is the Next Frontier

As I worked through these ideas, one thing became increasingly clear:

Even with comprehensive evaluation datasets, no test suite can realistically capture every scenario an autonomous agent will encounter in the real world.

This is why modern evaluation frameworks place so much emphasis on tracing and metrics. The same signals used to evaluate agents during testing can also be used to understand how they behave in production.

What I find particularly interesting is that these capabilities are not just useful for testing. They are foundational building blocks for systems that can learn from their own behavior.

That's a topic I'll explore in future articles, but it starts with a simple idea:

Before a system can learn from its behavior, we first need to be able to measure it.

Key Takeaways

• Traditional software testing breaks down with LLM-driven agents due to probabilistic behavior
• Agent evaluation requires measuring across 6+ dimensions (outcome, operational, reliability, quality, governance, behavioral)
• Three primary measurement approaches: mathematical metrics, specialized models, and LLM-as-a-Judge
• Traditional testing remains critical for deterministic system layers
• Modern frameworks converge on tracing, metrics, and persistent test datasets
• The future lies in observability-driven development where testing metrics inform production behavior

Originally published on Substack

The promise of AI agents is flexibility. They can solve problems we never anticipated. The challenge is control. Traditional testing can't capture every scenario an agent will encounter. That's why modern evaluation frameworks are moving toward observability: continuous measurement of agent behavior across outcome, reliability, cost, and safety dimensions.

We're building Agentplane to provide the infrastructure layer for this shift. TokenOps is our FinOps control plane that enforces token budgets and governance policies across agent delegation trees, giving you real-time cost attribution and risk enforcement. Chronicle captures execution traces and behavioral signals from every agent run, turning observability data into actionable insights for improvement and compliance.

Together, they close the gap between testing and production: measure your agents during development, enforce policies at runtime, and learn from real-world behavior.