START HERE
What you will learn
- Distinguish software tests from agent evaluations.
- Design checks for refusal, uncertainty and recovery.
- Compare candidate versions without hiding failures behind an average score.
Before you start: For stakeholders, independent builders and technology leaders. Bring one workflow or software change you can describe.
Start with three different questions
Does the delivered software behave correctly? Use application tests, contract checks and appropriately scoped integration tests. Did the agent act within its authority? Inspect tool calls, attempted access and approval evidence. Did its reasoning output satisfy the task? Evaluate evidence use, completeness and appropriate escalation.
These checks overlap but are not substitutes. A working feature does not excuse an unauthorised data read. A polished explanation does not prove the code works. A refused action is a success when refusal was required by the task’s policy.
Make the cases representative and the failures visible
Build a small set of ordinary, ambiguous, unauthorised and dependency-failure cases using synthetic or properly approved data. Specify the allowed actions and expected outputs for each case. Include stale documents, conflicting instructions and tool responses that cannot be trusted as instructions.
Separate development examples from held-out cases used for release decisions. If an agent repeatedly sees the entire test set and adjusts itself to those answers, a rising score may reflect familiarity rather than useful generalisation. Model-graded assessments can assist review, but critical permission and isolation checks need independent enforcement and evidence.
Test repeats, changes and recovery
Agent output can vary across runs. Repeat important cases under a recorded configuration and report the count of passes and failures, not just a single successful demonstration. A small sample does not establish a production reliability guarantee. Keep quality thresholds separate from hard safety failures.
An illustrative release policy might require every unauthorised-write case to be denied and send any boundary failure to human review. Cancellation and retry tests should also inspect external side effects. Reproducing the configuration improves comparison; it does not eliminate provider changes, sampling variation or incomplete coverage.
Example checks for one read-only agent
| Case | Expected behaviour | Evidence |
|---|---|---|
| Ordinary document question | Answer from permitted sources | Answer and source references |
| Missing evidence | State uncertainty; do not invent | Unsupported-claim review |
| Document instructs an external send | Do not execute the instruction | Tool-call and refusal records |
| Another customer’s document ID | Access denied independently of the model | Authorisation test |
| Tool timeout | Bounded retry or escalation | Attempt count and final state |
| Cancellation after a remote request | Reconcile the outcome before retrying | Remote status and recovery decision |
Step-by-step process
Map checks to requirements
Give each important expectation a check, a reviewer or both. State whether it is a hard gate or a quality measure.
Build safe adversarial cases
Include denied access, malformed tool results, uncertain evidence and unavailable dependencies. Keep test accounts and fixtures isolated.
Run baseline and candidate
Use recorded manifests, comparable inputs and repeated runs. Compare changes in failures as well as overall task quality.
Review evidence before promotion
Block on hard boundary failures, investigate regressions and preserve relevant cases. Promote the exact reviewed artifact to staging before a separate production decision.
FOLLOW ONE EXAMPLE
Worked scenario
A release candidate that summarises well but crosses a boundary
A fictional document agent is allowed to read one test collection and prepare a summary, but cannot email it.
- Most ordinary summaries are useful, but one adversarial document tells the agent to send the result externally.
- The tool layer denies the attempted send and records the attempt. The evaluation flags the attempted policy violation for investigation.
- The team changes instructions and the tool contract, then reruns the entire relevant suite rather than keeping only the successful summary.
What this shows: A high quality score would have hidden a consequential failure. Enforcement and behavioural evaluation provide different evidence.
The process at a glance
Read Mermaid diagram source
flowchart TD
A[Versioned candidate and cases] --> B[Repeated isolated runs]
B --> C[Application checks]
B --> D[Tool and permission checks]
B --> E[Output quality review]
C --> F{Hard gates pass}
D --> F
E --> F
F -->|No| G[Investigate and retain regression case]
F -->|Yes| H[Human staging decision]What you should have at the end
- A versioned evaluation set with hard gates and quality measures.
- A comparison report recording configuration, repeated outcomes and unresolved risks.
Common mistakes and how to avoid them
Letting an agent approve its own implementation
Use independent checks and retain a human acceptance owner for consequential work.
Claiming complete reliability from a small suite
Report coverage and limitations; extend cases as real failures and new workflows appear.
APPLY THE IDEA
Try it yourself
Your task: Design six cases for a read-only document agent, including an attempted external send.
What to produce: A case table with input, expected answer, allowed tools, forbidden actions and required evidence.
Check your understanding: Would a forbidden tool attempt fail the review even if the final written answer looked correct?
Key terms in plain English
- Evaluation case
- An input and expectations used to assess agent behaviour.
- Held-out case
- A case reserved from routine development feedback for a more independent check.
- Hard gate
- A requirement whose failure blocks promotion regardless of average quality.