An AI evaluation question set is a collection of realistic requests with a clear explanation of an acceptable response. Ask your agency to help build it before polishing a demonstration. Otherwise, everyone can agree that an answer sounds good while disagreeing about whether it is useful or supported.
Related service: AI automation development- 01Collect representative requests
- 02Label expected evidence
- 03Separate development and acceptance cases
- 04Run and review results
SECTION 01
Represent the work, including inconvenient cases
Collect ordinary questions, incomplete requests, outdated terminology, requests outside the agreed scope, and questions that need permission. A list containing only clean FAQs will favor a demonstration over the actual workflow. Note the source version and user role needed for each case so the same question is not judged against different conditions.
Ask for coverage by task rather than a single total count. Ten variations of a welcome message do not replace a missing account-access test. NIST's framework supports evaluating risks in context; your test categories should reflect the consequences of the task, not a universal accuracy target.
SECTION 02
Illustrative example: a delivery-policy assistant
Imagine a retailer asking whether a parcel can arrive before a holiday. One test supplies a destination but no order details; another includes an authenticated order with a carrier status. The public assistant should explain the approved policy and request the missing context. It should not invent a guaranteed arrival date.
For each test, record the allowed facts, a source reference, the required caveat or handoff, and any forbidden disclosure. Accept several natural phrasings when they convey the same meaning. If reviewers disagree, clarify the rubric instead of teaching the model one arbitrary sentence. This example is a planning exercise, not a reported project result.
SECTION 03
Buy a maintainable test asset, not a one-time score
Keep a held-back set that is not repeatedly used to tune prompts. Add newly discovered failures to a regression set, while preserving older cases that still describe supported behavior. Record model, prompt, source and integration versions with each run. Microsoft's RAG guidance distinguishes retrieval from answer generation; label which stage caused a failure.
At handoff, request the editable cases, reviewer notes, result history and instructions for rerunning tests. Demonstrate an intentionally unsupported answer being caught. Agree who updates expectations when a policy changes and which failures block release. The extra labeling work is useful only if the owner can maintain it after the agency leaves.
SECTION 04
Frequently asked questions
How many evaluation questions do we need?
Enough to represent the agreed task and its important exceptions. Start by covering distinct behaviors and user roles; increase examples where outcomes vary rather than chasing a universal count.
Can AI grade the answers?
It can assist comparison, but calibrate its judgments against human-reviewed cases. Keep authoritative review for consequential, disputed or permission-sensitive outcomes.
Should we accept one accuracy percentage?
No. Break results down by task and failure type. A strong average can hide unacceptable private-data disclosure or missing handoff behavior.
PRIMARY REFERENCES
Sources and further reading
These references cover the standards, platforms, or published prices discussed in the guide. Worked examples and checklists are our editorial guidance.
- AI Risk Management FrameworkNIST
- Retrieval-augmented generationMicrosoft Learn
EDITORIAL METHOD
About this guide
We use AI to assist with drafting and editing. Catapult AI Work is responsible for the published content. Examples illustrate possible approaches; they are not client case studies unless identified as such.
Budget examples are not Catapult package prices. Check linked provider pages for current fees and plan limits before making a purchase.
Read the editorial policy