Context: the model is one control among many

An agentic product usually combines a model with system instructions, user context, retrieval, memory, tool definitions, credentials, and application-side approval logic. A review that looks only at model outputs can miss the controls that actually determine whether an unsafe suggestion becomes an unsafe action.

The useful unit of analysis is the action path: what information enters, how instructions are prioritized, which capability is selected, what authority is attached, and what happens after the tool returns.

Agent security is largely the security of delegated authority under uncertain interpretation.

A four-part evaluation framework

1. Authority

List each tool and the identity used when it runs. Determine whether access is bound to the current user, a shared service account, or a broader system privilege. Review whether the tool can read, modify, publish, purchase, send, or delete.

2. Instruction flow

Trace where instructions can originate: system prompts, user messages, retrieved documents, web pages, tool output, memory, and other agents. Test whether untrusted content can become an instruction and whether the application distinguishes data from commands.

3. Data flow

Map sensitive inputs available to the model and tools. Check whether one user, tenant, task, or tool can influence data returned to another. Include logs, traces, embeddings, cache entries, and conversation memory.

4. Containment

Assume an incorrect action will eventually be proposed. Evaluate confirmation steps, transaction limits, dry-run behavior, reversibility, audit records, and emergency shutdown paths.

Testing in practice

Begin with ordinary user goals and identify the minimum permissions needed. Then introduce conflicting instructions through each untrusted channel. Vary the state of the application: authenticated versus unauthenticated, fresh versus long-running memory, direct user input versus retrieved content, and low-impact versus irreversible tools.

A strong finding does more than show that the model produced undesirable text. It demonstrates a reliable path from attacker-controlled input to a security-relevant decision or action, identifies the control that failed, and explains the conditions required to reproduce it.

Takeaways

  • Evaluate the complete action path, not the model in isolation.
  • Bind tool authority to the initiating user and current task wherever possible.
  • Treat retrieved content and tool output as untrusted data.
  • Require stronger confirmation as impact and irreversibility increase.
  • Design traces so investigators can reconstruct why an action occurred.