A successful demonstration shows that an agent can work once. An evaluation checks when it works, when it fails, and how serious those failures are. Start with examples of real tasks, then add the difficult cases your team expects to encounter.
Create the evaluation plan before launch and before optimizing prompts. NIST describes testing, evaluation, verification, and validation as evidence that AI systems meet organizational goals while limiting negative impacts. OpenAI's eval guidance similarly recommends task-specific evaluations and continuous evaluation. The checklist below turns those principles into production acceptance work.
Related service: AI & Automation services- 01Define the task, user, environment, authority, success criteria, and prohibited outcomes.
- 02Build versioned representative, edge, adversarial, permission, and failure test sets.
- 03Select deterministic checks, expert rubrics, model graders, and end-to-end outcome checks.
- 04Evaluate quality, retrieval, tools, security, humans, performance, cost, and recovery by category.
- 05Set severity-based acceptance thresholds and non-negotiable launch blockers.
- 06Run shadow and limited-release evaluations with real operational feedback.
- 07Continuously evaluate changes and sample production outcomes with privacy-aware monitoring.
SECTION 01
Write the evaluation contract before building the test
Describe the agent as an operating contract: who may request what objective, which data it may use, which tools and actions are permitted, what a complete result contains, when it must ask for help, and which outcomes are prohibited. Include the environment and consequence because the same answer can be harmless in a sandbox and damaging when connected to production tools.
Assign owners for task correctness, process policy, source data, security, privacy, technical operation, and release acceptance. Agree how disagreements will be resolved. A rubric that experts interpret differently cannot produce a stable launch gate until the underlying policy or task definition is clarified.
| Field | Definition | Example evidence |
|---|---|---|
| Task | One bounded objective and completion condition | Approved record created from a valid request |
| Authority | Allowed reads, proposals, writes, spending, and communication | Tool and field allowlist plus approval policy |
| Success | Quality, safety, performance, cost, and operational criteria | Thresholds by case type and severity |
| Prohibited outcome | Result that must block launch or trigger pause | Cross-tenant access or unapproved transaction |
| Fallback | How uncertain, failed, or unavailable work continues | Exception queue with owner and manual procedure |
| Change trigger | Component changes requiring regression evaluation | Model, prompt, corpus, permission, tool, or policy version |
SECTION 02
Build a representative and risk-based evaluation dataset
Sample real work across common categories, user roles, input formats, languages where supported, source versions, workflow states, and time periods. De-identify or secure sensitive records according to the test environment. Keep a held-out set for final comparison and prevent training or prompt examples from quietly absorbing every test case.
Add difficult cases deliberately: missing information, conflicting evidence, unsupported requests, ambiguous identity, inaccessible records, corrupted files, long input, duplicate events, tool timeouts, stale sources, and known historical failures. Add adversarial cases for direct and indirect prompt injection, secret requests, privilege escalation, data exfiltration, unsafe tool arguments, excessive loops, and resource exhaustion.
SECTION 03
Measure task quality with more than one score
Use deterministic graders for exact fields, schemas, totals, allowed categories, citations, and state transitions. Use expert rubrics for completeness, relevance, evidence use, reasoning where reviewable, escalation quality, and policy adherence. A model grader can help at scale, but calibrate it against human judgments and retain human review for disputed or consequential cases.
Report results by case category, consequence, and failure type. Exact-match accuracy may be useful for an extraction field but weak for a support resolution. Measure false positive and false negative costs separately. For multi-step work, score whether the final task is correctly completed as well as each critical intermediate step.
Correctness
The outcome, fields, action, and state match the expected task result.
Groundedness
Material claims are supported by accessible approved evidence or tool results.
Completeness
Required steps, fields, constraints, and follow-up are present.
Calibration
The agent asks for help or says it lacks evidence when the task cannot be completed safely.
Consistency
Equivalent cases receive materially equivalent treatment across repeated runs.
SECTION 04
Test retrieval, tools, state, and permissions independently
For retrieval, measure whether the correct source is permitted, available, retrieved, ranked, cited, and actually supports the response. Test obsolete and conflicting documents, no-answer conditions, and deletion. Do not give generation credit for an answer that was correct by chance when the required source was never retrieved.
For tools, test selection, validated arguments, user and tenant scope, return handling, timeouts, retries, duplicate prevention, partial failure, and reconciliation with the system of record. Bind approvals to the exact action and arguments reviewed. Confirm the agent cannot call hidden or disallowed functions by naming them in a prompt or retrieved document.
| Component | Pass evidence | Critical failure example |
|---|---|---|
| Retrieval | Expected accessible evidence is returned and cited | Cross-tenant or disallowed document retrieved |
| Planner | Permitted steps satisfy the objective within limits | Loops, invents a tool, or ignores a stop condition |
| Tool selection | Correct allowlisted tool for the task | Chooses a write or send tool unnecessarily |
| Arguments | Schema-valid, policy-valid, and bound to user authority | Injects untrusted content into executable or privileged fields |
| State | Resumes, retries, and ends without duplicate effects | Repeats a transaction after timeout |
| Approval | Reviewer sees evidence and exact proposed action | Approved action changes before execution |
SECTION 05
Run security, privacy, and abuse tests against the full workflow
OWASP's agentic guidance identifies risks that arise when systems can plan and act through tools. Test direct instructions that conflict with policy and indirect instructions hidden in documents, messages, webpages, or tool responses. Verify that untrusted content cannot change the system policy, expand permissions, expose secrets, or cause an unapproved external effect.
Review data minimization, tenant isolation, sensitive-data handling, logs, retention, deletion, model-provider settings, secrets, supply-chain dependencies, and incident response. Apply rate, spend, step, time, and tool limits. Test the kill switch and access revocation. Red teaming supports but does not replace a threat model, secure engineering, deterministic controls, and monitoring.
SECTION 06
Evaluate the human workflow, reliability, latency, and cost
Measure whether reviewers can understand the source, proposed result, validation state, and consequence quickly enough to make a real decision. Track approval, correction, rejection, escalation, review time, disagreement, and automation bias. If users routinely approve without reading or redo the task from scratch, the review design is not functioning.
Load-test realistic concurrency and input size. Simulate rate limits, slow providers, unavailable tools, database failures, and retries. Track end-to-end latency percentiles, timeout rate, duplicate effects, recovery time, model and infrastructure usage, human minutes, and cost per correctly completed task. Performance and cost failures can change user behavior and therefore quality risk.
| Measure | Definition | Why it matters |
|---|---|---|
| Correct completion | Accepted outcome completed end to end | Prevents partial model success from being counted as business success |
| Review effort | Human minutes per reviewed and corrected item | Reveals hidden operating cost |
| Exception age | Time unresolved work remains in the queue | Shows whether fallback is viable |
| Latency distribution | End-to-end percentiles by case type | Averages hide poor experience and timeout risk |
| Duplicate effect | Repeated message, record, or transaction | Tests retry and state safety |
| Unit cost | Technical plus human operating cost per correct outcome | Supports capacity and ROI decisions |
SECTION 07
Set launch gates and continuous-evaluation triggers
Define thresholds by severity. A small number of formatting defects may be acceptable with correction; one verified cross-tenant disclosure or unapproved transaction can be a launch blocker. Require sign-off from business, technical, security, and operational owners for the categories they own. Document known limitations and the exact initial user, data, tool, and authority boundary.
After launch, sample outcomes, monitor critical controls, review exceptions and corrections, and maintain traceability from an incident to component versions. Rerun the relevant frozen sets whenever prompts, models, retrieval indexes, source policies, tools, permissions, validators, or workflow steps change. Add new production failures as regression cases while retaining historical versions for comparison.
SECTION 08
Frequently asked questions
What should be tested before an AI agent goes to production?
Test task correctness, groundedness, retrieval, tool selection and arguments, identity, permissions, prompt injection, sensitive data, limits, state and retries, human review, latency, load, cost, monitoring, fallback, rollback, and disablement.
How many test cases does an AI agent need?
There is no universal number. The set needs enough coverage across common, important rare, edge, adversarial, permission, tool, and failure categories to support each launch decision. Track coverage and failure discovery, not a vanity case count.
Can an LLM grade another AI agent?
A model grader can help with scalable rubric-based review, but calibrate it against expert judgments and use deterministic checks where possible. Consequential or disputed cases still need accountable human evaluation.
When should AI agent evaluations be rerun?
Rerun relevant regressions after changes to the model, prompt, data, retrieval index, tool, permission, validator, policy, workflow, or provider behavior, and when production monitoring reveals a new failure category.
PRIMARY REFERENCES
Sources and further reading
These references cover the standards, platforms, or published prices discussed in the guide. Worked examples and checklists are our editorial guidance.
- AI Risk Management FrameworkNational Institute of Standards and Technology
- NIST AI Resource CenterNational Institute of Standards and Technology
- AI measurement and evaluationNational Institute of Standards and Technology
- Working with evalsOpenAI
- Agentic AI – Threats and MitigationsOWASP GenAI Security Project
- OWASP Top 10 for LLM and GenAIOWASP GenAI Security Project
EDITORIAL METHOD
About this guide
We use AI to assist with drafting and editing. Catapult AI Work is responsible for the published content. Examples illustrate possible approaches; they are not client case studies unless identified as such.
Budget examples are not Catapult package prices. Check linked provider pages for current fees and plan limits before making a purchase.
Read the editorial policy