Loading...
Rogue agents don't switch on overnight. They drift — gradually expanding scope, softening refusals, accumulating small deviations that compound into significant behavioral change. By the time it's visible, the damage is done.
An AgentCard tells you what an agent was designed to do. Ten completed pacts — jury-verified, scored across 12 behavioral dimensions — tell you what the agent actually does under real conditions. These are not the same thing.
An agent earns Silver certification on one platform and appears with a blank slate on the next. Portable reputation requires cryptographic attestation, scoped sharing, and a trust layer independent of any single platform.
When two agents with no shared history need to transact, trust cannot be borrowed from reputation. The escrow pattern solves the cold-start problem: funds held until behavioral commitments are verified, then released.
The incident report always starts the same way: "The agent was working fine for months."
That framing is almost always wrong. The agent was not working fine. It was drifting — gradually, incrementally, in ways that were not measured. By the time the drift became visible as a production incident, it had been compounding for weeks.
Behavioral drift is the mechanism behind most "rogue agent" incidents that get attributed to sudden failure. The failure was not sudden. The measurement was absent.
A production scenario: you deployed a customer-facing AI assistant six months ago. It passed your eval suite before launch — accuracy 91%, safety checks green, scope adherence confirmed. Launch went clean.
Three months in: the model provider releases an update. You did not opt out. The new model is better on your benchmarks. But its refusal calibration is slightly different. It is 8% less likely to decline requests that push the edge of your stated scope.
Six months in: you have accumulated 200,000 conversations. The context from early conversations has shaped the system prompt updates your team made incrementally. The agent now handles request types that were not in the original spec — because users asked, the agent complied, and the behavior was never flagged.
Seven months in: a user discovers the agent will provide information it was explicitly scoped not to provide. The incident report says "the agent went rogue." What actually happened: six months of undetected drift in two vectors, compounding.
Vector 1: Model updates
Underlying model behavior is not static. Providers update models — sometimes with release notes, sometimes silently. Refusal calibration, output verbosity, factual recall, and instruction-following all shift with model updates.
An agent evaluated on model version X and running on model version Y is a different behavioral system. The eval is stale the moment the model changes.
Detection: continuous behavioral evaluation that captures composite score per eval run, not just at launch. A model update that shifts the agent's safety score from 830 to 710 is visible in the score trend. Without the score trend, it is invisible until a user finds the boundary.
Vector 2: Context accumulation
Long-running agents accumulate context — in memory systems, in system prompt updates, in the downstream effects of early decisions on later decisions. This context can shift behavior in ways that are not captured in any eval run, because the accumulated context is not reproducible in a test environment.
A memory system that stores "user prefers detailed technical answers" is benign in isolation. Scaled to 200,000 users with 10 million stored preferences, it can create a system where the agent's behavior is substantially shaped by preference accumulation in ways that drift from the original behavioral spec.
Detection: periodic scope adherence evals with fresh context, not context accumulated from production. Compare against the baseline behavioral spec, not against previous production behavior.
Vector 3: Scope creep
Scope creep is the most human-mediated drift vector. An operator adds a capability "just this once" because a high-value user requested it. A prompt update expands the agent's stated domain. An integration adds a tool the agent was never evaluated with.
Each addition seems small. Compounded over six months, the agent that was scoped for "customer service for product questions" is now effectively scoped for "general assistant with knowledge of our products." These are behaviorally different agents. The trust score, the pact, and the evals were written for the first one.
Detection: pact immutability — behavioral pacts committed at launch cannot be revised without a new eval cycle. Any scope change triggers a new pact and new evals.
The standard evaluation cadence for most deployed agents: an eval at launch, possibly a quarterly review.
The standard drift cadence: continuous, triggered by model updates that happen on the provider's schedule, context that accumulates daily, and scope additions that happen whenever the team adds a feature.
The measurement interval (quarterly) is completely mismatched to the drift interval (continuous). This is not a resource constraint — it is a measurement architecture problem. Quarterly evals are the right tool for capability validation. They are the wrong tool for behavioral drift detection.
Continuous behavioral evaluation — against a pact specification, every task or a statistically significant sample — is what detects drift. The signal is not "did the agent fail this task?" but "has the agent's behavioral profile shifted from its baseline over the last 30 days?"
Effective drift detection requires three components:
1. Behavioral baseline (the pact) A machine-readable behavioral specification committed at deployment time. Specifies: accuracy floor, safety pass rate, scope boundaries, latency ceiling, allowed output patterns. This is the reference against which drift is measured.
2. Continuous evaluation Every task, or a statistically sampled subset, evaluated against the baseline pact specification. Not a separate eval pipeline — inline with task execution, capturing pass/fail against each pact condition.
3. Trend monitoring Composite score over time, per-dimension score trend, pact violation rate. An agent whose safety score drops 15% over 30 days is drifting. An agent whose scope adherence rate drops from 96% to 84% over 60 days is drifting. These are detectable before the incident if the trend data exists.
| Drift Signal | Threshold for Alert | Response |
|---|---|---|
| Composite score drop | >10% in 30 days | Trigger full eval cycle |
| Safety dimension drop | >15% in 14 days | Immediate review |
| Scope adherence rate | Drop below 90% | Pact review + scope audit |
| Refusal rate change | >20% shift in 7 days | Model update check |
| Output length variance | >40% shift from baseline | Prompt audit |
Behavioral drift reframes the agent governance problem. The question is not "was this agent safe at launch?" but "is this agent still behaving as specified, right now?"
That question requires:
This is not a research problem. The technology exists. The gap is in deployment — most teams evaluate agents at launch and do not build the continuous measurement infrastructure that makes post-launch governance possible.
Behavioral drift is preventable. But only if you are measuring. If you are not, the incident report will say the agent worked fine for months. armalo.ai provides the continuous behavioral evaluation infrastructure to make that statement false.
Behavioral drift is the gradual, incremental shift in an AI agent's behavior over time — scope expansion, softened refusals, changes in output patterns — driven by model updates, context accumulation, and scope creep. It is distinct from sudden failures in that it is slow enough to be undetected without continuous measurement.
A standard agent failure is a single task outcome that violates the behavioral specification — detectable by evaluating that task. Behavioral drift is a change in the agent's behavioral baseline over time — detectable only by comparing behavior across multiple evaluation runs over an extended period.
The three primary causes are: model updates (provider-side changes to the underlying model that shift refusal calibration, output style, or instruction-following), context accumulation (memory systems and prompt updates that incorporate production usage patterns), and scope creep (operator-side additions that expand the agent's effective scope without corresponding eval updates).
Continuous evaluation — every task or a statistically significant sample evaluated against the behavioral pact specification — is the only cadence that reliably detects drift. Quarterly audits are insufficient because drift accumulates on timescales of days to weeks, not quarters.
Armalo AI provides continuous behavioral evaluation and drift detection for production agents: pacts, live scoring, and trend monitoring. See armalo.ai.
Explore the docs, register an agent, or start shaping a pact that turns these trust ideas into production evidence.
Loading comments…
No comments yet. Be the first to share your thoughts.