Integrations
Setting Latency Budgets for AI Features
Break AI response time into measurable stages. Set budgets for retrieval, generation, tools, and retries so slow requests lead to useful engineering decisions.
Users experience the total time required to complete their task, not just the model’s generation speed. An AI feature may authenticate, retrieve evidence, rerank it, call tools, validate output, and save a result. Google’s SRE guidance treats latency as a core service signal. Applying that idea well requires measuring the complete path and deciding which waits the product can tolerate.
Measure stages and user-visible milestones
Track time to the first useful feedback, time to the first generated content when relevant, and time to a complete usable result. Break down server work into spans for retrieval, model calls, validation, and persistence. A fast first token can hide a slow or failed completion, so both milestones matter.
Look at distributions and slow cases instead of only an average. Group requests by task type and input size. A short extraction and a long research job should not share an unexplained latency target. Separate successful and failed requests, because a quick error does not represent a good user experience.
Allocate a budget to the task
Imagine a hypothetical sales portal generating a short meeting brief. The product team sets an illustrative completion target based on how people use the page. Engineers then allocate portions to retrieving account records, generating the brief, and validating required sections. Those allocations are hypotheses to test, not universal recommended timings.
If retrieval consumes most of the budget, changing the generation model may have little effect. If independent data sources are fetched sequentially, safe parallelization may help. If the task genuinely needs extended analysis, move it into a background job with visible progress and a return path rather than forcing it into a fragile request window.
Keep timeout behavior coherent
Set an overall deadline and ensure nested retries cannot exceed it indefinitely. When a user cancels or navigates away, decide which work should stop and which durable job should continue. Bound tool calls and preserve any state needed to explain partial completion.
Test cold starts, slow dependencies, large inputs, and concurrent demand. Review whether a quality stage such as reranking earns its added delay. Avoid removing validation merely because it is easy to measure. The objective is a timely, useful outcome with predictable failure behavior, so latency changes should be reviewed alongside correctness, evidence quality, and user correction effort.
