Testing & quality
Create a human review rubric for AI outputs
Human review works better when people share a concrete standard. Define dimensions, severity, and examples that turn opinions into actionable product feedback.
Human review is valuable when an AI output requires judgment, but a rating such as good or bad is difficult to act on. A rubric gives reviewers a common language. For a hypothetical proposal-drafting tool, a response can be readable yet include an unapproved delivery promise. Evaluating those dimensions separately helps the team improve the right behavior instead of debating whether the paragraph feels professional.
Start from the task and its risks
List the decisions the reviewer should make. A proposal draft might need factual consistency, completeness, appropriate tone, and adherence to approved commercial information. Define each dimension in ordinary language. Keep the list short enough for consistent use. If two dimensions regularly produce the same explanation, consider combining them. If one dimension hides a serious failure inside a broad judgment, split that failure into a separate requirement.
Use anchored ratings
Describe what each rating means with an example. For completeness, an acceptable draft might include the agreed scope and the next step; a minor issue might omit a useful background detail; a major issue might omit an essential dependency. Specify whether the reviewer should judge only the visible evidence. Reviewers should not silently supply knowledge from another meeting and then penalize the model for failing to know it.
Calibrate before reviewing at scale
Ask two reviewers to assess the same small batch independently. Compare their explanations, especially disagreements about severity. Resolve unclear wording in the rubric before assigning a larger collection. Keep an adjudication note for recurring disputes. Agreement does not prove that the rubric measures the right thing, so include the product owner in deciding whether the dimensions reflect actual user outcomes and meaningful business constraints.
Turn findings into engineering work
Require a short evidence-based note for important failures. An observation such as invented a delivery date is easier to investigate than sounds overconfident. Group findings by likely cause without forcing reviewers to diagnose implementation details. Report the number of reviewed examples and how they were selected. A small convenience sample should inform a decision, but it should not be presented as a precise estimate of all production behavior.
