START HERE
What you will learn
- Create a simple scoring checklist before testing.
- Compare an AI-assisted process with your existing method.
- Explain the limits of a small trial without overstating its results.
Before you start: You need a clearly described task and a few examples of acceptable finished work.
What are you evaluating?
Separate the draft from the finished outcome. If the job is to produce a reliable comparison brief, a fast but inaccurate draft is not a successful result. You need to know how long it takes to reach an acceptable final brief, including preparation and corrections.
Write the acceptance criteria before looking at the new system’s answers. Acceptance criteria are the conditions a result must meet: all required questions answered, claims supported, no missing owner or date, and no unauthorised disclosure. Choose criteria that fit the task rather than copying a generic accuracy score.
Use examples that reveal differences
Include ordinary inputs, awkward inputs and questions that cannot be answered from the available information. A system that always answers may look helpful until you test whether it can recognise missing evidence.
Keep some examples aside while adjusting the system. If you repeatedly rewrite the prompt to pass the same few cases, the final score mainly describes those cases. A small trial is evidence about the examples tested, not proof that every future task will work.
Step-by-step process
Record the baseline
Use your existing method on a small set of tasks. Record total time and the quality of the final result. Keep the task conditions reasonably comparable.
Define the checklist and serious failures
A serious failure might be an invented commitment or a confidential passage shown to the wrong person. Track these separately; do not let a high average score hide them.
Run the AI-assisted version
Keep a note of the tool, prompt version, input and output. Include review and rework in the timing. Do not quietly remove difficult examples from the results.
Compare the results by task type
Ask where the approach helps and where it fails. It may be useful for simple summaries but unsuitable for conflicting source documents.
Decide the next release boundary
You might allow use for one task type, revise the process or stop. If you proceed, continue checking a sample of real outcomes and revisit the evaluation when the task changes.
FOLLOW ONE EXAMPLE
Worked scenario
Testing a weekly research brief
For this fictional exercise, you test ten briefs. Every brief must cover the requested questions, support its claims and identify missing information.
- The existing method takes 40 minutes per finished brief. The AI-assisted trial takes 28 minutes to prepare and review a draft, but two drafts still need corrections. The two timings do not yet describe equally finished work.
- Two AI-assisted briefs contain unsupported cost claims. Your acceptance rule says unsupported commercial claims must be corrected before use.
- You investigate why those two cases failed and retest the change on fresh examples. Record the total effort again for acceptable finished briefs. Do not describe the initial trial as ready for unrestricted use.
What this shows: It is too early to call the 12-minute difference a saving, because the correction effort is not yet known. The quality check prevents an attractive draft-time figure from becoming an unsupported rollout claim.
The process at a glance
Read Mermaid diagram source
flowchart LR
A[Define acceptable work] --> B[Select test examples]
B --> C[Compare existing and AI-assisted methods]
C --> D[Record quality and effort]
D --> E{Serious failures resolved?}
E -->|Yes| F[Allow a limited use case]
E -->|No| G[Revise and retest]What you should have at the end
- A repeatable checklist, with serious failures clearly identified.
- A comparison that includes preparation and review time.
- A limited, evidence-based decision about where the workflow may be used.
Common mistakes and how to avoid them
Judging only the most impressive answer
Evaluate the planned set of cases, including the awkward ones.
Averaging away a serious failure
Keep consequential errors visible even when most outputs look good.
Calling a tiny sample a proven saving
State the sample, conditions and uncertainty. Larger or different work may behave differently.
APPLY THE IDEA
Try it yourself
Your task: Choose five examples of one task, including one with missing information. Write the acceptance checklist before running them.
What to produce: A results table with input type, pass/fail reasons, review effort and the decision to continue or revise.
Check your understanding: Could someone repeat your comparison and understand why you made the release decision?
Key terms in plain English
- Evaluation set
- The examples used to assess how a system performs.
- Rubric
- A written scoring guide that makes quality judgements more consistent.
- Baseline
- The existing method used for comparison.
- Regression
- A previously acceptable behaviour that gets worse after a change.