START HERE

What you will learn

  • Locate the trust boundaries around a model, documents and tools.
  • Distinguish a malicious instruction from the application authority to act.
  • Define practical tests for data access, generated output and tool permissions.

Before you start: The RAG and human-review guides explain retrieval and approval. This article covers defensive design and testing on systems you are authorised to assess.

Reference edition: OWASP Top 10 for LLM Applications 2025. Sources checked 2026-11-11.

The application is more than the model

A large language model (LLM) can generate text, while the surrounding application controls documents and tools. Consider a document assistant with an email tool. The model receives text, but the application also chooses accessible documents, constructs requests and executes permitted tool calls. A secure design must examine those surrounding operations.

The 2025 guidance includes risks such as Prompt Injection and Excessive Agency. The first concerns attempts to change behaviour through instructions in input; the second concerns giving a model-backed system more ability to act than it needs. A message can ask for an action without being entitled to authorise it.

Examine data, output and action boundaries separately

A retrieved document might contain hostile instructions. Private information might reach an answer through an overly broad search. Generated text might become unsafe executable input if another component trusts it without suitable handling. An agent might request a tool action beyond the user’s permissions.

These are related concerns but require different controls. Better wording in a system prompt can help state intentions; it cannot replace permission enforcement by the application, safe handling of outputs or a real approval step for consequential actions. This guide applies selected risks, rather than claiming to reproduce or fully test the entire official list.

Step-by-step process

  1. Map the data and tool paths

    List where user text and retrieved material enter, which documents are eligible, and which tools can read or change anything.

  2. Keep untrusted content separate from authority

    Treat retrieved material as evidence to inspect, not a source of permissions. Enforce access checks before sensitive information reaches the model.

  3. Constrain tool operations outside the model

    Use narrowly defined operations and check the user, target and allowed arguments. A model’s request is not itself permission to send a message or alter a system. Bind approval to the specific operation and target; a changed request needs a fresh decision.

  4. Handle generated output for its destination

    A draft shown as text and a value inserted into a query or command have different handling needs. Do not automatically execute generated strings as trusted instructions.

  5. Test refusals, permissions and resource limits

    Use benign test fixtures containing conflicting instructions, inaccessible document requests and disallowed tool operations. Check the application’s actual decisions and include limits on repeated calls or expensive work.

FOLLOW ONE EXAMPLE

Worked scenario

An assistant encounters an instruction inside a document

In an authorised test, a fictional document assistant retrieves a fixture containing an instruction to send its findings to an unapproved address. No real confidential data is used.

  1. The retrieved passage is treated as document content, not as permission to send an email.
  2. If the model requests the email operation, the application checks the allowed recipient and required approval. The unapproved operation is rejected.
  3. The test checks the tool decision and any resulting action, not just whether the model said it would refuse. The fixture becomes a regression case for later changes.

What this shows: The defence is demonstrated at the action boundary. A persuasive instruction in a document does not gain the authority to send information.

The process at a glance

The model can propose an operation. Application permission checks and any required human approval determine whether it may happen.
ProcessDecisionHuman checkpoint
Read Mermaid diagram source
flowchart TD
 A[Question and untrusted document text] --> B[Model proposes an answer or action]
 B --> C{Application permits this operation?}
 C -->|No| D[Reject and record the decision]
 C -->|Yes| E{Human approval required?}
 E -->|Yes| F[Human reviews target and effect]
 E -->|No| G[Run the constrained operation]
 F --> H{Approval granted?}
 H -->|Yes| G
 H -->|No| D

What you should have at the end

  • A map of untrusted inputs, private data and permitted tools.
  • Application-enforced boundaries around retrieval and consequential actions.
  • Recorded tests showing what happened when a disallowed operation was requested.

Common mistakes and how to avoid them

Relying only on “ignore malicious instructions”

Keep independent permission and tool checks even if prompts describe the intended behaviour.

Trusting an answer because it contains citations

A citation does not establish that the document was authorised or the claim supported.

Giving a general shell or send-anywhere tool for convenience

Define the smallest operation needed and the approval it requires.

APPLY THE IDEA

Try it yourself

Your task: Design a synthetic document fixture that asks your test assistant for an operation outside its allowed scope. Do not include secrets or target another person’s system.

What to produce: A defensive test record showing the expected rejection, actual tool decision and whether any action occurred.

Check your understanding: Would the application still deny the operation if the model requested it confidently?

Key terms in plain English

Trust boundary
A point where data or requests cross between different permission or trust contexts.
Prompt injection
Untrusted instructions attempting to redirect model or application behaviour.
Agency
The ability of a system to select or perform actions through tools.
Output handling
Preparing generated material safely for the component that receives it.

Sources and further reading