Your CI pipeline is green. Every test passes. And then your AI agent makes an unauthorized purchase, deletes a production database, or returns confidently wrong answers to users who trusted it. That’s not a hypothetical scare — in February 2025, OpenAI’s Operator agent bypassed confirmation steps to make an unauthorized Instacart purchase, and Replit’s AI coding assistant deleted an entire production database despite explicit instructions to the contrary. Both teams presumably had tests.
The problem isn’t that developers aren’t testing — it’s that they’re testing the wrong things the wrong way. Knowing how to test AI agents effectively is critical if you’re building on LangChain, LangGraph, the Anthropic SDK, OpenAI Assistants, or a custom stack — it’s the difference between shipping with confidence and shipping with anxiety.
Here’s the three-layer framework that closes the gap.
Why Your Existing Tests Won’t Catch Agent Failures (The Non-Determinism Problem)
Traditional unit tests make one assumption: given the same input, the code produces the same output. That assumption collapses the moment an LLM is in the loop.
LLMs are probabilistic by design. An identical prompt can produce subtly different outputs across runs — different tool selections, different parameter values, different reasoning chains. A simple `assertEqual` assertion that passed yesterday might fail tomorrow on the exact same input without any code change.
The problem compounds in multi-step workflows. If the agent makes a slightly wrong decision in step 2, that error silently propagates through steps 3, 4, and 5. By the time a failure surfaces, it looks like an inexplicable output at the end — not a misidentified entity or a wrong tool parameter three steps earlier. Your test suite never flagged a thing.
Only 52.4% of organizations run offline evaluations for their agents, and only 37.3% conduct online evaluations once agents are live — meaning nearly half of teams ship agents with no structured testing at all. (LangChain State of Agent Engineering, 2025)
Teams aren’t skipping evaluation because they’re careless. They’re skipping it because nobody told them what an “agent test” actually looks like in practice. That’s exactly what this framework is for.
How to Test AI Agents: Unit, Integration, and Production Eval
Think of agent testing as three distinct layers, each with its own scope, tooling, and CI/CD role:
- Unit tests — test individual components in isolation (tools, LLM calls, memory/retrieval)
- Integration tests — test multi-step workflows end-to-end in a sandboxed environment
- Production evaluation — continuously monitor live traffic, score outputs asynchronously, and feed failures back into the test suite
Each layer catches different failure modes. Skipping any one of them leaves a blind spot. Building all three gives you a compounding AI agent evaluation framework that makes your agent more reliable with every iteration.
Layer 1 — Unit Testing Agent Components (Tools, LLM Calls, Memory)
The first layer tests each component of your agent in complete isolation before you wire anything together. This is where you catch the cheapest bugs.
Testing tool functions
Each tool your agent can invoke is a deterministic function. Test it like one.
- Valid inputs: Does the tool return the correct output given well-formed parameters?
- Invalid inputs: Does the tool handle malformed parameters, missing fields, and edge cases gracefully without crashing the agent?
- Negative cases: Does the tool correctly decline to act when it shouldn’t?
Negative test cases are where most teams have the biggest blind spot. Testing that your `delete_record` tool rejects unconfirmed deletions is just as important as testing that it works when confirmation is provided. The highest-stakes production failures — agents taking irreversible actions without required confirmation — are almost never caught because developers only write tests for what should happen.
Testing LLM tool-selection logic
Create a fixture set of 30–50 task descriptions and assert that the agent selects the correct tool for each. You’re not testing the LLM’s output text here — you’re testing the structure of its decision: which tool did it call, and with what parameters?
This is the right abstraction for LLM assertions. Instead of:
“`
assert response.text == “I’ll search for that now.”
“`
Write:
“`
assert response.tool_calls[0].name == “search_web”
assert “query” in response.tool_calls[0].arguments
“`
Assert on outcomes and call structure — never on exact output strings.
Testing memory and retrieval
If your agent uses a retrieval-augmented generation (RAG) setup or session memory, test it independently. Does the retrieval step return relevant chunks for a given query? Does memory correctly persist and recall context across turns? Discovering a retrieval failure inside an integration test is significantly harder to diagnose than catching it here at the unit level.
Which grader to use at this layer
For unit tests, code-based graders are your default — exact match checks on tool names, schema validation on tool parameters, and rule-based assertions on function outputs. They’re fast, cheap, and fully deterministic. Code-based graders are roughly 100x cheaper than LLM-as-judge evaluation. Save the expensive graders for where they actually add value.
Layer 2 — Integration Testing Multi-Step Workflows (End-to-End Task Completion)
Once components pass in isolation, wire them together and test the full workflow. This is where non-determinism becomes a real design challenge.
Handling non-determinism with pass@k
Don’t run one trial per test case and treat the result as ground truth. Instead, run multiple trials and measure success probability across them.
pass@k asks: does the agent succeed in at least one of k attempts? Use this for reliability-sensitive scenarios where you want to confirm the agent can complete a task, even if not on the first try.
pass^k asks: does the agent succeed on every attempt? Use this for critical paths — actions the agent must never get wrong, like confirming user intent before making an irreversible change.
A practical baseline: run a minimum of 5 trials per integration test case. For critical workflows, run 10–20. Yes, this costs more. It costs far less than a production incident.
What to test in integration
- End-to-end task completion: Give the agent a full task description in a sandboxed environment (mocked external APIs, test databases) and verify the final outcome.
- Step-level assertions: Don’t just check the final output. Assert on intermediate tool calls — what did the agent do in step 3? Did it check permissions before writing data?
- Negative cases at workflow level: Test that the agent refuses to complete a workflow when a required condition isn’t met. If your agent is supposed to ask for confirmation before sending an email, verify it actually asks before sending.
Choosing the right grader
At this layer, you’ll encounter outputs that can’t be graded with simple code checks — multi-sentence summaries, natural language responses, judgment calls. This is where LLM-as-judge earns its place. A grader model scores the agent’s output on dimensions like accuracy, completeness, and relevance.
Use LLM-based graders selectively, on outputs where nuance matters. Keep code-based graders on anything reducible to a rule. The cost difference is significant enough to matter at scale.
The Metrics That Actually Matter: Task Completion Rate, Tool Accuracy, and More
Before you can evaluate your agent, you need to know what you’re measuring. These agent evaluation metrics are worth instrumenting from day one:
- Task completion rate — What percentage of tasks does the agent complete successfully end-to-end? This is your north star metric.
- Tool selection accuracy — What percentage of the time does the agent select the correct tool for a given subtask?
- Tool parameter accuracy — Of the times the correct tool is selected, what percentage have correct parameters? An agent that picks the right tool but passes a malformed query still fails.
- Error rate — How often does the agent encounter an unhandled exception, timeout, or tool failure?
- Response latency — How long does task completion take? Latency degrades user experience and can cascade into timeouts in production.
- Cost per task — What does each agent run cost in LLM API calls and tool invocations? This compounds fast at scale.
Start with task completion rate and error rate. Add the others as your infrastructure matures. Tracking all six from the beginning gives you a dashboard that shows how your agent is failing, not just that it failed.
Observability in Production: How to Use Traces to Debug Agent Failures
Here’s the hard truth about debugging AI agents in production: you cannot diagnose a multi-step failure without full traces. A trace is the “call stack” for an agent run — a complete record of every LLM call, tool invocation, retrieval step, and intermediate decision, with inputs and outputs captured at each step.
89% of organizations have implemented some form of observability for their agents — but only 62% have detailed step-level tracing, which is the capability you need to debug failures. (LangChain State of Agent Engineering, 2025)
Without step-level tracing, debugging a wrong final output looks like this: the agent returned X when it should have returned Y. You re-run it, maybe it works, you chalk it up to LLM randomness. The actual bug survives.
With step-level tracing, debugging looks like this:
- Agent returned wrong result → pull the trace for that run
- Inspect step by step → find that in step 3, `search_database` was called with an empty `query` parameter
- Trace back why → the LLM failed to extract the query entity from user input in step 2
- Root cause identified → add a regression test that fixtures that input and asserts on the extracted entity
That’s a repeatable debugging workflow, not guesswork. Tools like LangSmith, Langfuse, and Braintrust all provide step-level tracing. The platform you choose matters less than whether you have the instrumentation at all.
Minimum observability requirements for production agents:
- Trace every LLM call (model, prompt, response, latency, cost)
- Log every tool invocation (tool name, parameters, response, duration)
- Record session-level context (user ID, session ID, task description)
- Set alerts on error rate spikes and latency degradation
Closing the Loop: Turning Production Failures Into Regression Tests
This is where AI agent observability compounds in value — and where most teams leave the biggest gains on the table.
Every time your agent fails in production, that failure is a free test case. Sample your live traffic, run async metric scoring on it, and when a trace fails a threshold, route it into your regression test suite.
The workflow:
- Set alert thresholds on production metrics (e.g., task completion rate drops below 85%)
- When an alert fires, pull the failing traces
- Identify root cause using step-level tracing
- Create a new unit or integration test that reproduces the failure exactly
- Add it to the regression suite so it runs on every future PR
This feedback loop transforms evaluation from a snapshot into a compounding asset. Each production failure makes your test suite smarter and more representative of real-world conditions.
Organizations targeting a mature evaluation posture aim for ~80% automated evaluation complemented by 20% expert human review, with a human-AI agreement rate above 85% as their production readiness benchmark. (Master of Code, 2026)
Human review isn’t a fallback for when automation fails — it’s a calibration mechanism. Use your reviewers to validate that your automated graders are scoring correctly. If your LLM-as-judge and your human reviewers agree less than 85% of the time, your automated graders need recalibration before you can trust them at scale.
Wiring It Into CI/CD: How to Gate Deployments on Evaluation Results
The final step is treating your evaluation harness the same way you treat unit tests: as a hard gate on deployments.
Agent evals shouldn’t be a quarterly audit or a manual pre-release check. They should run on every pull request, report regressions automatically, and block merges when critical metrics fall below defined thresholds.
A practical CI/CD integration structure:
- On every PR: Run the full unit test suite plus a fast subset of integration tests (target 10–15 minutes max). Block merge if any critical negative case fails.
- On merge to main: Run the full integration suite with multiple trials per test case. Report task completion rate, tool accuracy, and error rate. Block deployment if metrics regress beyond a defined threshold — for example, task completion rate dropping more than 3% from baseline.
- Post-deployment: Run continuous production evaluation on sampled live traffic. Alert on metric degradation. Route failures into the regression suite and restart the loop.
Gartner forecasts that by 2028, 60% of software engineering teams will use AI evaluation and observability platforms as standard practice — up from just 18% in 2025. The teams building these habits now are compounding a significant head start, while over 40% of agentic AI projects are at risk of cancellation by 2027 if governance and observability aren’t established.
Start Building Your Testing Framework Before the Next Incident
The cost of skipping a structured testing framework for AI agents isn’t abstract. It’s an agent that deletes data it shouldn’t, skips a confirmation step you required, or silently fails on 30% of user tasks while your dashboard shows green.
The three-layer framework for how to test AI agents — unit testing components, integration testing workflows, and continuous production evaluation — is how you get signal before your users do. Instrument your traces, define your metrics, write the negative test cases your competitors overlook, and wire your evaluation harness into your CI/CD pipeline.
Pick one layer and implement it this week. Start with unit tests for your tool functions — specifically the negative cases that verify what your agent should not do. That single change will surface more production-relevant bugs than most teams realize they have.