Testing & quality
Choose golden examples for an AI product
Golden examples turn vague quality expectations into concrete product behavior. Select representative cases and explain why each response is acceptable.
A golden example is an approved illustration of what good behavior looks like for a specific task. It is especially useful when stakeholders agree that an answer should be clear but disagree about what clear means. Imagine a service desk that wants an AI assistant to summarize incoming tickets. A small set of carefully reviewed examples can establish what belongs in a summary and what must remain uncertain.
Choose examples that teach a decision
Start with cases that expose an important choice. A routine ticket might contain a clear problem and a device name. Another may mention two unrelated issues. A third may quote an earlier message that no longer describes the current state. Each approved summary should show how the product handles that choice. Ten examples of essentially identical requests provide less direction than a smaller, deliberately varied set.
Annotate the reasoning behind approval
Store the approved result alongside a short explanation. Identify the facts that must appear, details that can be omitted, and assumptions the assistant must avoid. For the quoted-message example, explain why an old error code should not become the current diagnosis. These notes help a new reviewer understand the standard without treating the original phrasing as sacred. They also reveal where the product policy still needs a decision.
Include acceptable alternatives
A good response often has more than one valid form. Preserve alternatives when they satisfy the same requirements, especially for summaries and customer-facing drafts. Mark exact requirements separately: a case identifier or date may require strict matching even when the rest is flexible. Include an unacceptable near miss with a reason when helpful. The contrast should clarify a meaningful error, such as confusing the reported symptom with a confirmed cause.
Give the examples an owner
Golden examples age when product behavior changes. Assign a reviewer and record the policy or workflow revision each example represents. Before reusing them in a prompt, confirm that their data is appropriate for that context. Before using them as evaluation material, check whether the system has already been tuned closely around them. Maintain additional unseen cases so the team can assess general behavior beyond the examples everyone knows by heart.
