BlogTesting & quality

Testing & quality

Use contract tests around AI integrations

AI outputs still cross ordinary software boundaries. Contract tests protect schemas, error handling, and action requests when prompts or providers change.

An AI integration is part of a larger application, and that application expects specific data. A model may produce a plausible classification while omitting the identifier needed by the next screen. Contract tests check these boundaries independently of whether a response sounds intelligent. Imagine a lead-routing tool whose AI component proposes a department and a reason. The receiving system must still validate that department and handle an unusable result.

Specify the application contract

Describe the fields, allowed values, optional values, and failure states the integration exposes. Distinguish a missing field from a deliberately empty value. Decide what happens when the model cannot classify a request. For lead routing, an explicit needs_review state may be more useful than guessing a department. Document the contract at the application boundary so the frontend does not need to understand provider-specific response details.

Test malformed and incomplete responses

Create fixtures for valid output, unknown categories, missing required fields, invalid types, and extra unexpected content. Check how the adapter handles each case. Validation should occur before a response can trigger an action or update business records. A schema-valid response is not necessarily a correct classification, so keep these tests distinct from quality evaluations. Each layer should answer a clear question about a different kind of failure.

Include operational failure contracts

Test timeouts, rejected requests, interrupted responses, and temporary upstream errors. Verify the application's error format and whether retrying is appropriate. If the integration can initiate an external action, test duplicate requests and the mechanism that prevents repeated execution. Use deterministic substitutes for most contract tests so they remain fast and repeatable. Keep a smaller live integration check to detect changes that a simulated response cannot reveal.

Version the adapter deliberately

Treat the adapter as the place where provider details become stable application behavior. When a provider or prompt changes, run the same contract suite before exposing the candidate to users. If the application contract must change, coordinate the consumers and migration plan. In the lead-routing example, adding a new department also requires checking dropdowns, reporting filters, permissions, and fallback handling rather than simply accepting another string.