AI engineering
Designing Useful Fallbacks for AI Service Failures
Plan how your AI feature behaves during timeouts, invalid responses, and provider outages. Useful fallbacks preserve user work and make uncertainty clear.
An AI feature depends on services that can slow down, reject requests, or return unusable output. A fallback is the product behavior chosen when the preferred path cannot complete. Microsoft’s circuit-breaker guidance describes stopping calls to a persistently failing dependency. The user-facing design must go further and decide what useful, honest result remains available.
Classify failures before choosing a fallback
Separate transient transport errors, rate limiting, invalid structured output, missing evidence, and authorization failures. These conditions should not all trigger the same retry. A permission denial must not be bypassed by switching providers. A lack of supporting evidence is not necessarily a provider outage.
Give each processing stage a time budget and bounded retry policy. Record the original failure and any fallback used. If a secondary path has different capabilities, validate that it can satisfy the task contract before invoking it. The ability to produce text is insufficient for a workflow requiring a particular schema or approved data handling.
Preserve a useful user journey
Imagine a hypothetical knowledge portal where answer generation times out after search succeeds. The interface can still show relevant documents and preserve the question for retry. That is often more useful than an empty error screen. Label the state clearly so the user understands that the search results are available but the generated answer did not complete.
For a drafting feature, save the user’s input and any completed draft without pretending a partial response is final. For an action-taking assistant, distinguish an action that definitely failed from one with unknown status. Check the operation record before repeating a write, especially when the connection failed after the remote service may have accepted it.
Rehearse degraded operation
Test a slow first response, a broken stream, malformed output, and a dependency that fails repeatedly. Check whether retries create a burst of additional load and whether cancellation stops unnecessary work. Circuit breakers and backoff can help, but they need application-specific thresholds and recovery behavior.
Define what operators should see: error categories, affected task types, fallback frequency, and the age of queued work. Verify recovery as well as failure. Once the preferred service returns, the product should resume normal operation without abandoning pending jobs or repeating completed actions. Keep fallback quality visible in evaluation rather than counting every completed response as equally successful.
