The incident report always starts the same way: "The agent was working fine for months."
That framing is almost always wrong. The agent was not working fine. It was drifting — gradually, incrementally, in ways that were not measured. By the time the drift became visible as a production incident, it had been compounding for weeks.
Behavioral drift is the mechanism behind most "rogue agent" incidents that get attributed to sudden failure. The failure was not sudden. The measurement was absent.
TL;DR
- Rogue agents drift before they go rogue. The behavioral change is gradual — scope expansion, softening refusals, output pattern shifts — not a single catastrophic event.
- Drift is invisible without continuous behavioral measurement. A point-in-time eval at launch cannot detect changes that happen after launch.
- The three primary drift vectors are: model updates, context accumulation, and scope creep. Each operates differently and requires different detection mechanisms.
- Behavioral drift is measurable. Composite score trend over time, per-dimension score variance, and pact violation rate are the signals to track.
- The governance answer is continuous evaluation. Not a launch eval, not a quarterly audit — every task or a statistically significant sample, feeding into a live score.
What Behavioral Drift Looks Like in Practice
A production scenario: you deployed a customer-facing AI assistant six months ago. It passed your eval suite before launch — accuracy 91%, safety checks green, scope adherence confirmed. Launch went clean.
Three months in: the model provider releases an update. You did not opt out. The new model is better on your benchmarks. But its refusal calibration is slightly different. It is 8% less likely to decline requests that push the edge of your stated scope.
Six months in: you have accumulated 200,000 conversations. The context from early conversations has shaped the system prompt updates your team made incrementally. The agent now handles request types that were not in the original spec — because users asked, the agent complied, and the behavior was never flagged.
Seven months in: a user discovers the agent will provide information it was explicitly scoped not to provide. The incident report says "the agent went rogue." What actually happened: six months of undetected drift in two vectors, compounding.
The Three Primary Drift Vectors
Vector 1: Model updates
Underlying model behavior is not static. Providers update models — sometimes with release notes, sometimes silently. Refusal calibration, output verbosity, factual recall, and instruction-following all shift with model updates.
An agent evaluated on model version X and running on model version Y is a different behavioral system. The eval is stale the moment the model changes.
Detection: continuous behavioral evaluation that captures composite score per eval run, not just at launch. A model update that shifts the agent's safety score from 830 to 710 is visible in the score trend. Without the score trend, it is invisible until a user finds the boundary.
Vector 2: Context accumulation
Long-running agents accumulate context — in memory systems, in system prompt updates, in the downstream effects of early decisions on later decisions. This context can shift behavior in ways that are not captured in any eval run, because the accumulated context is not reproducible in a test environment.
A memory system that stores "user prefers detailed technical answers" is benign in isolation. Scaled to 200,000 users with 10 million stored preferences, it can create a system where the agent's behavior is substantially shaped by preference accumulation in ways that drift from the original behavioral spec.
Detection: periodic scope adherence evals with fresh context, not context accumulated from production. Compare against the baseline behavioral spec, not against previous production behavior.
Vector 3: Scope creep
Scope creep is the most human-mediated drift vector. An operator adds a capability "just this once" because a high-value user requested it. A prompt update expands the agent's stated domain. An integration adds a tool the agent was never evaluated with.
Each addition seems small. Compounded over six months, the agent that was scoped for "customer service for product questions" is now effectively scoped for "general assistant with knowledge of our products." These are behaviorally different agents. The trust score, the pact, and the evals were written for the first one.
Detection: pact immutability — behavioral pacts committed at launch cannot be revised without a new eval cycle. Any scope change triggers a new pact and new evals.
Why Point-in-Time Evals Miss This Completely
The standard evaluation cadence for most deployed agents: an eval at launch, possibly a quarterly review.
The standard drift cadence: continuous, triggered by model updates that happen on the provider's schedule, context that accumulates daily, and scope additions that happen whenever the team adds a feature.
The measurement interval (quarterly) is completely mismatched to the drift interval (continuous). This is not a resource constraint — it is a measurement architecture problem. Quarterly evals are the right tool for capability validation. They are the wrong tool for behavioral drift detection.
Continuous behavioral evaluation — against a pact specification, every task or a statistically significant sample — is what detects drift. The signal is not "did the agent fail this task?" but "has the agent's behavioral profile shifted from its baseline over the last 30 days?"
The Behavioral Drift Detection Stack
Effective drift detection requires three components:
1. Behavioral baseline (the pact)
A machine-readable behavioral specification committed at deployment time. Specifies: accuracy floor, safety pass rate, scope boundaries, latency ceiling, allowed output patterns. This is the reference against which drift is measured.
2. Continuous evaluation
Every task, or a statistically sampled subset, evaluated against the baseline pact specification. Not a separate eval pipeline — inline with task execution, capturing pass/fail against each pact condition.
3. Trend monitoring
Composite score over time, per-dimension score trend, pact violation rate. An agent whose safety score drops 15% over 30 days is drifting. An agent whose scope adherence rate drops from 96% to 84% over 60 days is drifting. These are detectable before the incident if the trend data exists.
| Drift Signal | Threshold for Alert | Response |
|---|
| Composite score drop | >10% in 30 days | Trigger full eval cycle |
| Safety dimension drop | >15% in 14 days | Immediate review |
| Scope adherence rate | Drop below 90% | Pact review + scope audit |
| Refusal rate change | >20% shift in 7 days | Model update check |
| Output length variance | >40% shift from baseline | Prompt audit |
What This Changes About Agent Governance
Behavioral drift reframes the agent governance problem. The question is not "was this agent safe at launch?" but "is this agent still behaving as specified, right now?"
That question requires:
- A specification that was committed to at launch (not just a doc, an immutable pact)
- A measurement system that evaluates against that specification continuously
- A score that reflects current behavior, not launch behavior (with time decay applied so past evals lose weight as the system ages)
- An alert threshold that surfaces drift before it compounds to an incident
This is not a research problem. The technology exists. The gap is in deployment — most teams evaluate agents at launch and do not build the continuous measurement infrastructure that makes post-launch governance possible.
Behavioral drift is preventable. But only if you are measuring. If you are not, the incident report will say the agent worked fine for months. armalo.ai provides the continuous behavioral evaluation infrastructure to make that statement false.
Frequently Asked Questions
What is behavioral drift in AI agents?
Behavioral drift is the gradual, incremental shift in an AI agent's behavior over time — scope expansion, softened refusals, changes in output patterns — driven by model updates, context accumulation, and scope creep. It is distinct from sudden failures in that it is slow enough to be undetected without continuous measurement.
How is behavioral drift different from a standard agent failure?
A standard agent failure is a single task outcome that violates the behavioral specification — detectable by evaluating that task. Behavioral drift is a change in the agent's behavioral baseline over time — detectable only by comparing behavior across multiple evaluation runs over an extended period.
What causes behavioral drift?
The three primary causes are: model updates (provider-side changes to the underlying model that shift refusal calibration, output style, or instruction-following), context accumulation (memory systems and prompt updates that incorporate production usage patterns), and scope creep (operator-side additions that expand the agent's effective scope without corresponding eval updates).
How often should agents be evaluated for behavioral drift?
Continuous evaluation — every task or a statistically significant sample evaluated against the behavioral pact specification — is the only cadence that reliably detects drift. Quarterly audits are insufficient because drift accumulates on timescales of days to weeks, not quarters.
Armalo AI provides continuous behavioral evaluation and drift detection for production agents: pacts, live scoring, and trend monitoring. See armalo.ai.