Testing & quality
Build an AI test dataset that reflects real work
A useful AI test dataset captures routine requests, awkward exceptions, and clear success criteria. Build a collection that makes product decisions easier.
An AI feature needs a collection of examples that makes quality discussable. A few impressive demonstrations cannot tell you whether the product handles routine work, confusing requests, or missing information. Start with the decisions your users actually need to make. For a hypothetical equipment support assistant, the question is whether a technician can identify the right approved procedure, including when no suitable procedure exists.
Define the unit of evaluation
Choose one complete task for each example. Record the user request, the information available at that moment, the expected behavior, and any forbidden behavior. An equipment question might include a machine model and a manual excerpt. Its expected result could be a grounded explanation with a reference, or a request for the missing model number. Avoid scoring a response against information the system never received.
Sample situations deliberately
Create a coverage table before gathering examples. Rows can represent common equipment families; columns can represent straightforward questions, ambiguous names, outdated documents, and unsupported requests. Fill gaps using clearly labeled synthetic examples. These are design probes, not evidence of real customer frequency. Keep a separate field for where each example originated so future reviewers can distinguish observed problems from anticipated ones and prioritize appropriately.
Separate improvement from measurement
Use a working collection while adjusting prompts and retrieval. Reserve another collection to assess changes without repeatedly designing around its exact wording. Keep related variations together when splitting the data, because a paraphrase of the same question can give an overly reassuring result. Preserve difficult cases even after they pass. A previously fixed failure is valuable protection against a later regression.
Make the collection maintainable
Assign a person to approve expected outcomes and remove obsolete examples. Include an example identifier, category, review date, and reason for inclusion. Remove unnecessary personal information before storing material. When a source manual changes, review affected examples rather than silently retaining an old answer. Track coverage alongside pass rates: a large collection with one type of question can still leave an important workflow effectively untested. Write down disputed cases until the product owner resolves them.
