Search & knowledge
Evaluating Retrieval With Real Business Questions
Build a retrieval evaluation set that reveals missing evidence, weak rankings, stale sources, and permission mistakes before they become polished AI answers.
An answer can sound helpful while relying on the wrong evidence. Retrieval evaluation examines the documents and passages available to the generator before judging its prose. This separation helps teams distinguish search failures from generation failures. It also makes experimentation more useful: changing chunking, ranking, or filtering should have a visible effect on the evidence selected.
Write questions and evidence expectations together
Choose questions from the work the product is meant to support. For each question, identify acceptable source passages, required qualifiers, and cases where no answer should be produced. Avoid a dataset made entirely from document headings; users rarely phrase every question like the source author.
Include ordinary vocabulary, abbreviations, exact identifiers, and incomplete questions. Label the access context and source version as part of the case. A document can be relevant in general but wrong for a particular tenant, product edition, or effective date. Those distinctions should appear in the evaluation record rather than being left to memory.
Inspect both coverage and ordering
Consider a hypothetical field-service assistant asked about restarting a unit after a specific alarm. The correct procedure may appear in the candidate pool but below a generic restart guide. Measure whether the needed passage is retrieved at all and whether it appears in the final set supplied to generation.
Semantic reranking, as described by Elastic, reorders an existing candidate set. This distinction suggests a useful diagnostic: if the passage is absent, improve candidate retrieval; if it is present but buried, investigate ranking. Also inspect duplicated passages. A shortlist containing five copies of the same general advice has less useful coverage than its length suggests.
Turn evaluation results into engineering decisions
Group failures by cause and document type. A table extraction issue may affect only product catalogs, while a filter bug may affect every query for one organization. Prioritize changes by the importance of the affected workflow and the frequency of the failure, not just the ease of improving a score.
Keep a held-out set for checking changes after tuning. Review a sample manually, especially borderline judgments about whether a passage supports an answer. Add newly discovered failure cases over time without erasing the original baseline. Record the corpus snapshot and retrieval configuration for each run so comparisons remain meaningful when sources change.
