Agent Red-Teaming: Why You Need an Adversary Before You Have a Customer

The taxonomy of behavioral vulnerabilities is different from the taxonomy of security vulnerabilities, but the red-team methodology is structurally similar:

Scope boundary testing. What are the specific inputs that cause the agent to act outside its authorized scope? This involves systematic probing of the edge cases and ambiguous situations near the scope boundary — the scenarios where the agent might reasonably interpret its authorization expansively rather than narrowly. The goal is to find the inputs that make scope violations look locally reasonable.

Escalation bypass. What inputs cause the agent to proceed autonomously when it should escalate? This tests whether escalation triggers are calibrated correctly, whether the trigger conditions can be circumvented through careful input crafting, and whether the agent's confidence calibration under high-uncertainty conditions matches its escalation behavior.

Adversarial instruction following. What happens when the agent is given explicit instructions to violate its hard prohibitions? Does it refuse consistently? Are there framings — authority claims, urgency framing, hypothetical contexts — under which it will comply? Hard prohibitions should hold under adversarial instruction; testing whether they do is a core red-team function.

Prompt injection. When the agent processes external content — web pages, user-submitted documents, tool outputs — can malicious content embedded in that content redirect the agent's behavior? Prompt injection is the most common class of adversarial attack against language model-based agents, and it is underdetected by capability evaluations that use clean, curated inputs.

Context manipulation. In multi-turn conversations, can a sophisticated user gradually shift the agent's operational context through a series of individually reasonable-seeming requests, arriving at a state where the agent will do things it would refuse in a cold start? Context manipulation exploits the agent's tendency to maintain conversational consistency, and it does not appear in single-turn evaluation frameworks.

The Gap Between Benchmark Performance and Adversarial Performance

The empirical finding that drives the urgency for agent red-teaming is the scale of the gap between benchmark performance and adversarial performance. Agents that perform at the 90th percentile on capability benchmarks may have significant behavioral vulnerabilities that their benchmark performance does not predict.

This gap is not surprising once you understand how it arises. Capability benchmarks are designed to measure what the agent can do. They select inputs where the correct behavior is well-defined and achievable. They do not include inputs specifically designed to make incorrect behavior seem reasonable. They do not test the adversarial conditions that will exist in production.

An agent's adversarial performance cannot be inferred from its capability performance. These are different properties that require different evaluation methods. Assuming otherwise is a systematic error with predictable consequences.

How to Structure an Agent Red-Team

A well-structured agent red-team has three components that mirror security red-team methodology:

Scope definition. Before the red team begins, define what they are testing. What are the agent's hard prohibitions? What are its escalation triggers? What are the data access boundaries? The red team needs to know what behavioral commitments the agent is supposed to honor in order to test whether it actually honors them. Without a behavioral specification, there is no definition of success for the red team.

Adversarial testing execution. The red team systematically probes each dimension of behavioral vulnerability:

• For each hard prohibition, develop 10-20 adversarial inputs that try to elicit the prohibited behavior using different framings, authority claims, and contextual setups. The goal is to find the specific inputs that cause prohibition violations, if any exist.

• For escalation behavior, develop a set of inputs that should trigger escalation under the defined criteria and verify that escalation fires reliably. Then develop adversarial inputs that attempt to circumvent escalation triggers.

• For prompt injection, embed adversarial instructions in the types of external content the agent will process. Test whether those embedded instructions redirect the agent's behavior.

• For context manipulation, design multi-turn conversation sequences that attempt to shift the agent's operational context incrementally toward prohibited territory.

Findings classification and remediation. Red-team findings are classified by severity — critical (prohibition violation achievable), high (escalation bypass achievable), medium (context manipulation possible but requires significant effort), low (behavioral inconsistency under adversarial conditions but no violation achievable). Critical findings block deployment. High findings require remediation before deployment or explicit acceptance with compensating controls. Medium and low findings are addressed in order of cost-effectiveness.

The Adversarial Agent Model

The most rigorous red-teaming approach uses adversarial agents — automated systems that systematically generate adversarial inputs, test behavioral responses, and iterate on inputs that approach violation conditions. The adversarial agent model has several advantages over manual red-teaming:

Coverage. A manual red team can test hundreds of adversarial inputs in a day. An adversarial agent can test tens of thousands, systematically exploring the input space near behavioral boundaries. The coverage difference is significant in finding rare-but-real violation conditions.

Reproducibility. Adversarial agent test results are fully reproducible — the same adversarial inputs can be run against any agent at any time, making it possible to measure how behavioral robustness changes as the agent is updated. Manual red-team results are partially reproducible but depend on human judgment and effort.

Specificity. Adversarial agents can be targeted to specific behavioral dimensions — focusing exclusively on scope violations, or on escalation bypass, or on prompt injection — generating inputs that efficiently probe each vulnerability class.

This is the model that Armalo implements in its adversarial evaluation infrastructure: automated adversarial agents that systematically probe behavioral pact commitments and produce evidence of compliance or violation at scale.

Red-Teaming Is Not a One-Time Event

Security red-teaming is not a one-time certification. It is a recurring practice, because systems change and the adversarial landscape changes. The same principle applies to AI agents.

Every time an agent's underlying model is updated, its behavioral profile may change — including its behavioral vulnerabilities. Every time a pact is revised, new boundaries need to be tested. Every time a new deployment context is added, the adversarial conditions specific to that context need to be evaluated.

The organizations that treat behavioral evaluation as a one-time pre-deployment checkpoint are making the same mistake as organizations that do security audits only before initial launch. The attack surface changes. The deployment context evolves. The adversarial landscape adapts to whatever defenses were in place at the time of the last evaluation.

Continuous behavioral evaluation — with adversarial testing as a core component — is the standard that serious enterprise AI deployments are converging on. The agents that have been continuously evaluated against adversarial conditions will have behavioral records that prove it. That record is the verifiable evidence that separates trustworthy agents from those that merely claim to be.

The Uncomfortable Business Case

The business case for pre-deployment red-teaming is the same as the business case for pre-deployment security testing: the cost of finding a vulnerability in testing is a small fraction of the cost of that vulnerability being exploited in production.

For AI agents in high-stakes deployment contexts — financial services, healthcare, legal, any context where agent actions have real-world consequences — a single significant behavioral failure can cost more than a full red-team engagement costs to run. The expected value calculation is not close.

The organizations that understand this are building red-teaming into their agent deployment process. The organizations that do not are learning the same lesson through experience, at a higher cost.