AI engineering
Building Boundaries Against Prompt Injection
Treat retrieved documents and tool results as untrusted data. Reduce prompt-injection exposure with scoped tools, trusted policy checks, and adversarial tests.
Prompt injection occurs when content supplied to an AI system tries to redirect its behavior through instructions embedded in data. That content can arrive through documents, web pages, messages, or tool responses. OWASP’s prevention guidance recommends layered defenses. For product teams, the central architectural task is to prevent untrusted text from acquiring the authority of application policy or user authorization.
Identify the untrusted entry points
Map where external content enters the system and where it can influence a decision. Search snippets, uploaded files, repository comments, and connector records all deserve attention. A document that appears in a trusted storage service may still have been written by someone outside the current user’s authority boundary.
Keep application instructions and retrieved material clearly separated in the request structure. Label sources and preserve provenance. This helps the model interpret content, but formatting alone is not a complete defense. The service performing an action must validate permissions and arguments regardless of what the model says motivated the request.
Limit the consequences of a mistaken interpretation
Imagine a hypothetical research assistant reading a supplier document containing a paragraph that tells it to send private project notes to an external address. The document should supply facts about the supplier, not grant permission to transmit unrelated data. A narrowly scoped retrieval tool and a separate, authorized messaging path reduce the consequences of that attempted redirection.
Avoid giving a single tool unrestricted read and write access across systems. Constrain destinations, resource scope, and allowed operations in application code. Where consequential actions need review, show the actual target and content. Do not let an instruction found inside source material satisfy the approval requirement.
Test behavior across the full path
Create adversarial examples that resemble the material your product handles: a misleading PDF footnote, an instruction inside a support ticket, or a tool result claiming to be a system update. Test both direct leakage attempts and subtler requests to change the task, suppress evidence, or call an unnecessary tool.
Inspect tool attempts and retrieved context, not only final text. An assistant could produce a harmless-looking answer after already making an unsafe request. Keep incident records with the relevant source identifiers and attempted actions. Treat automated detection as one signal among several; a detector is not a substitute for restricted capabilities and independent authorization.
