An AI agent evaluation framework tests the complete workflow, not only the model response. Define expected outcomes, permitted actions, required evidence, failure severity, and human fallback. Build cases from normal work, edge cases, attacks, and prior incidents. Score deterministic facts with code, nuanced outputs with structured human rubrics, and use explicit release gates before expanding permissions or changing production versions.
Playbook
An AI agent evaluation framework tests the complete workflow, not only the model response. Define expected outcomes, permitted actions, required evidence, failure severity, and human fallback. Build cases from normal work, edge cases, attacks, and prior incidents. Score deterministic facts with code, nuanced outputs with structured human rubrics, and use explicit release gates before expanding permissions or changing production versions.
Scope and Evidence Status
This framework reflects sources available on 30 July 2026. It is recommended practice, not an official certification scheme.
The voluntary NIST Generative AI Profile recommends pre-deployment testing, red-teaming, documentation, monitoring, and evaluation matched to risk. NIST’s agentic evaluation probes explore automated verification of factual grounding and machine-readable trails connecting claims to evidence.
NIST’s automated benchmark guidance was still described as an initial public draft during this review period. Do not market conformance to a draft as certification.
The NCSC secure deployment guidance recommends effective security evaluation and red-teaming before release, while the OWASP agentic threats guide supplies adversarial patterns for autonomous systems.
Start With an Evaluation Contract
Before writing test cases, define:
• Workflow: the business process and user.
• Intended outcome: the external result that counts as success.
• Allowed actions: tools and operations the agent may use.
• Blocked actions: operations never allowed without a person.
• Required evidence: sources or records needed for a valid result.
• Human fallback: when and where the agent must stop.
• Failure severity: consequence categories.
• Release gate: evidence required to proceed.
Example:
The agent may classify an inbound support request, retrieve approved help-centre sources, draft an internal recommendation, and create a triage task. It may not close the ticket, change an entitlement, or send a sensitive response without approval.
This is testable. “Improve support with AI” is not.
The Eight Evaluation Dimensions
1. Task outcome
Did the workflow reach the correct business state?
• Correct classification or routing.
• Required fields completed.
• No duplicate object.
• Correct recipient and channel.
• Proper escalation.
• Final state reconciles with the source system.
Use deterministic checks where possible.
2. Factual grounding
Does every material claim have adequate support?
• Correct source retrieved.
• Source is current and authorised.
• Citation supports the claim.
• Conflicting sources are recognised.
• Missing evidence triggers escalation.
An answer can sound useful while citing the wrong account or obsolete policy.
3. Tool use and side effects
Test:
• Correct tool selection.
• Valid parameters.
• Least-privilege operation.
• Idempotency on retry.
• No unauthorised sequence.
• Accurate interpretation of tool response.
• Correct rollback or compensating action.
Simulate partial failures. Multi-step agents frequently encounter one successful action followed by one failed action.
4. Policy compliance
Create cases for:
• Sensitive data.
• Disallowed recipients.
• Pricing, legal, employment, or payment commitments.
• Missing human approval.
• Conflicting instructions.
• Low-confidence or incomplete context.
• Requests outside intended purpose.
The expected result can be safe refusal, escalation, or draft-only output.
5. Security and adversarial resilience
Include:
• Direct prompt injection.
• Instructions embedded in retrieved documents.
• Tool-description poisoning.
• Memory poisoning.
• Credential or secret extraction attempts.
• Malicious URLs and files.
• Requests to override policy or conceal actions.
• Infinite-loop and resource-exhaustion patterns.
OWASP guidance is a source of threat cases, not proof that a ten-item checklist makes an agent secure.
6. Human oversight
Test whether:
• The reviewer sees the relevant evidence and proposed action.
• Approval is required before execution.
• Rejection prevents the action.
• Edits are recorded.
• Escalation reaches an authorised person.
• Timeouts default safely.
• Review workload remains practical.
A nominal human-in-the-loop control can fail because the reviewer cannot see what matters.
7. Reliability and change
Run:
• Repeated trials of important cases.
• Model, prompt, and tool-version comparisons.
• High-volume and concurrency tests.
• Timeout and dependency failure tests.
• Long-context and multilingual cases where supported.
• Regression cases from every prior incident.
Nondeterminism means one pass is not enough for consequential cases.
8. Cost and operational value
Measure:
• Cost per successful case.
• Tool calls and steps per success.
• Human review minutes.
• Failure and retry cost.
• Latency to usable outcome.
• Comparison with the manual baseline.
An agent can pass quality tests and still be unsuitable for production economics.
Build the Test Set
Use five buckets:
1. Golden path: frequent, well-defined work.
2. Edge cases: missing fields, conflicting evidence, unusual formats.
3. Negative cases: requests the agent must refuse or escalate.
4. Adversarial cases: attacks on goals, tools, memory, and identity.
5. Regression cases: every material production defect or near miss.
Do not allow synthetic examples to dominate. Sample redacted production work with appropriate privacy controls. Record provenance and expected outcomes.
For each case, store:
• Case ID and version.
• Input and protected context references.
• Expected outcome.
• Allowed and prohibited actions.
• Required evidence.
• Severity if failed.
• Scoring method.
• Reviewer and rationale.
Choose the Right Scoring Method
Deterministic checks
Use code for object states, field values, permissions, duplicate detection, schema validation, citations, and approval ordering.
Structured human review
Use a rubric for usefulness, tone, completeness, material factual errors, and context-dependent judgement. Define examples for each score and sample reviewer agreement.
Model-based graders
LLM judges can scale review but introduce their own variability and bias. Validate them against expert-labelled cases, pin versions, record prompts, and do not use them as the only gate for high-impact failures.
Production outcome validation
Confirm whether the actual ticket, task, message, invoice, or CRM record reached the intended state. Self-reported agent success is insufficient.
Define Risk-Based Release Gates
Separate:
Safety and policy gates
• Zero unapproved high-impact actions in the test set.
• Zero secret disclosure in defined adversarial cases.
• Approval and rejection paths work.
• Rollback succeeds for mutable actions.
Quality gates
• Performance against the workflow’s accepted rubric.
• No regression in critical cases.
• Adequate grounding and source freshness.
Operational gates
• Tool reliability, latency, cost, and review load are acceptable.
• Monitoring and incident owners are active.
Do not copy these as universal numeric thresholds. Define internal values from consequence, volume, baseline, and fallback.
Use permission stages:
1. Offline evaluation.
2. Shadow mode.
3. Read-only.
4. Draft-only.
5. Low-risk action with sampling.
6. Higher-impact action with explicit approval.
Passing one stage permits a controlled next stage, not unlimited autonomy.
Implementation Steps
1. Write the evaluation contract.
2. Map failure modes and severity.
3. Collect representative cases.
4. Define expected outcomes and actions.
5. Automate deterministic checks.
6. Create and calibrate human rubrics.
7. Add OWASP-informed adversarial cases.
8. Run repeated tests.
9. Review failures by severity, not only average score.
10. Approve, constrain, or reject the release.
11. Monitor production and add regressions.
Pair this with the AI automation audit checklist before launch.
Evidence and Artifacts
• Evaluation contract.
• Versioned test set and provenance.
• Failure taxonomy and severity.
• Rubrics and labelled examples.
• Automated check code and results.
• Model, prompt, tool, policy, and workflow versions.
• Reviewer decisions and disagreement records.
• Release-gate report.
• Known limitations.
• Production-monitoring link.
• Regression case from each incident.
• Approval for permission-stage change.
Common Failure Modes
Only model outputs are scored. Tool and business state failures remain untested.
Average score hides a severe failure. A single unapproved payment matters more than many good summaries.
Test cases are too clean. Production includes missing, conflicting, and malicious context.
The same model grades itself. Correlated errors can look like agreement.
A new model ships without regression tests. General benchmark improvement does not guarantee workflow performance.
Human review is assumed, not tested. The queue may be too slow or evidence may be missing.
No case represents prior incidents. The system can repeat known failures.
SMB Evaluation Checklist
• [ ] Define the intended business outcome.
• [ ] List allowed and prohibited actions.
• [ ] Set required evidence and fallback.
• [ ] Classify failure severity.
• [ ] Include real, redacted workflow examples.
• [ ] Add edge, negative, adversarial, and regression cases.
• [ ] Check tool side effects deterministically.
• [ ] Test approval, rejection, timeout, and rollback.
• [ ] Validate model graders against human labels.
• [ ] Repeat critical cases for variability.
• [ ] Compare cost and review time with the baseline.
• [ ] Set risk-based release gates.
• [ ] Expand permissions in stages.
• [ ] Add every material incident to regression tests.
FAQs
What should an AI agent evaluation measure?
Measure task outcome, grounding, tool use, policy compliance, security, human oversight, reliability, cost, and actual business state.
How many test cases does an SMB need?
There is no universal number. Cover frequent work, material edge cases, every prohibited action, credible attacks, and prior incidents. Risk coverage matters more than a large undifferentiated count.
Can an LLM judge replace human evaluation?
Not safely in every case. Use model graders for scale after validating them against labelled examples. Keep deterministic and expert review for high-impact requirements.
When is an agent ready for autonomous action?
Only after it passes relevant gates, works in controlled production stages, has monitoring and rollback, and the residual risk is explicitly accepted. Some actions should always require approval.
Should evaluations run after launch?
Yes. Run regression evaluations after model, prompt, tool, policy, data, or workflow changes and add cases from production failures.
Get a 20-Minute AI Workflow Audit
AI Operator can convert one workflow into an evaluation contract, test set, severity model, and staged release gate.