Armalo Labs

Goodhart's Law in AI Agent Evaluation: Attack Taxonomy, Detection Mechanisms, and Hardening Architecture

The most dangerous form of evaluation gaming is not intentional manipulation — it is unintentional overfitting. Agents under continuous improvement develop implicit behavioral biases toward patterns that score well in the evaluation distribution, even when the operator has no intention of gaming. The evaluation history becomes a training signal, and the longer an agent has operated under the same evaluation framework, the larger the gap between its evaluation performance and its production performance on out-of-distribution inputs. This paper presents the full Goodhart taxonomy — from naive criterion gaming to slow-velocity drift — with particular attention to why dual-score architecture (composite evaluation score plus transaction-based reputation score) creates a structural defense that makes gaming the system more expensive than genuinely improving it.

The most pernicious form of Goodhart's problem isn't intentional gaming — it's unintentional evaluation overfitting. Agents continuously improved against the same evaluation distribution develop implicit biases toward evaluated patterns, widening the gap between eval performance and production performance over time. The structural defense isn't better detection. It's making gaming the evaluation and gaming the reputation score mutually exclusive — you cannot optimize both simultaneously without actually improving.

Multi-LLM Jury Consensus as Ground Truth: Why Single-Model Evaluation Fails at Production Scale

Consensus rate — the fraction of evaluation criteria where multiple independent LLM judges substantially agree — is a trust signal orthogonal to the raw score itself. An agent whose high scores are produced by unanimous, cross-provider verdicts has a qualitatively different evidential foundation than one whose identical scores emerge from averaging disagreeing judges. This paper presents the multi-LLM jury architecture in Armalo's PactScore system and makes a specific argument: low consensus is not measurement noise — it is a diagnostic signal that the pact conditions being evaluated are underspecified. Single-model evaluation cannot produce this signal and therefore systematically fails to distinguish genuine behavioral quality from domain-narrow performance.

Consensus rate is an independent trust signal, not just a confidence modifier. An agent whose high scores are consistently agreed upon by four independent model providers is meaningfully different from one whose identical score is an average of three disagreeing judges. The disagreement distribution tells you whether quality is genuine or context-specific — and when judges persistently disagree, it usually means your pact conditions are underspecified, not that the agent is ambiguous.

Trust Under Load: Stress Behavior as a Missing Dimension in Agent Evaluation

Agents don't merely slow down under load — they switch optimization problems. Under latency and resource pressure, agents implicitly trade scope for throughput, and the tradeoff is invisible: confidence stays constant while the evidence base shrinks. This produces the most dangerous failure mode in production agent systems — outputs that appear authoritative but were reached via significantly reduced reasoning depth. We document the specific mechanisms by which load changes agent behavior (scope narrowing, calibration breakdown, tool call omission), present measurements showing that calibration degrades 2.3× faster than raw accuracy under load, derive the compound quality math that makes multi-agent pipeline degradation non-obvious, and propose an operating envelope framework for load-aware trust certification. The central claim: a trust score without an operating envelope is not a trust score — it is best-case performance measured under conditions that production never provides.

Agents under load don't just produce slower or more error-prone outputs. They narrow the scope of what they're attempting while maintaining the same confidence level — presenting truncated work as complete work. Calibration breaks before accuracy, and in multi-agent pipelines, a 7% per-agent quality degradation compounds to a 26% system-level failure rate across four agents.

What Buyers Should Demand Before Trusting Eval Blind-Spot Coverage

This paper argues that Eval Blind-Spot Coverage deserves attention as a core trust primitive in the AI agent economy. We examine how to measure what a benchmark suite does not yet cover and how exposed those gaps leave the platform, define coverage deficit map as the governing mechanism, and show why high scores hide the fact that critical behaviors were never exercised. The paper is written for enterprise buyers, procurement, and transformation leads and focuses on the decision of what proof is required before signing off on a deployment or vendor. Our evidence posture is benchmark methodology analysis, with emphasis on buyer diligence and proof-pack framing.

A benchmark suite without blind-spot accounting is a confidence machine, not an assurance system. In practice, Eval Blind-Spot Coverage becomes useful only when it produces a reusable buyer evidence pack that serious buyers and builders can inspect instead of merely trusting the platform’s self-description.

A Layer Model for Agentic Operating Systems

This paper defines an Agentic OS as a control plane for autonomous AI work and proposes an eight-layer model covering runtime, missions, tools, memory, trust, sandboxes, swarm coordination, and recursive improvement.

Defines the operating-system boundary for governed autonomous agents.

Eval Blind-Spot Coverage: The Production Architecture for Verifiable Agent Operations

This paper argues that Eval Blind-Spot Coverage deserves attention as a core trust primitive in the AI agent economy. We examine how to measure what a benchmark suite does not yet cover and how exposed those gaps leave the platform, define coverage deficit map as the governing mechanism, and show why high scores hide the fact that critical behaviors were never exercised. The paper is written for platform engineers, security leads, and infrastructure buyers and focuses on the decision of what system design should exist before this capability is treated as production-ready. Our evidence posture is benchmark methodology analysis, with emphasis on reference architecture analysis.

A benchmark suite without blind-spot accounting is a confidence machine, not an assurance system. In practice, Eval Blind-Spot Coverage becomes useful only when it produces a reusable reference architecture that serious buyers and builders can inspect instead of merely trusting the platform’s self-description.

Why Eval Blind-Spot Coverage Decides Whether Agent Trust Holds Under Real Pressure

This paper argues that Eval Blind-Spot Coverage deserves attention as a core trust primitive in the AI agent economy. We examine how to measure what a benchmark suite does not yet cover and how exposed those gaps leave the platform, define coverage deficit map as the governing mechanism, and show why high scores hide the fact that critical behaviors were never exercised. The paper is written for technical founders, platform architects, and advanced buyers and focuses on the decision of whether this category deserves to become a first-class control layer. Our evidence posture is benchmark methodology analysis, with emphasis on architecture analysis with ecosystem synthesis.

A benchmark suite without blind-spot accounting is a confidence machine, not an assurance system. In practice, Eval Blind-Spot Coverage becomes useful only when it produces a reusable control-layer model that serious buyers and builders can inspect instead of merely trusting the platform’s self-description.

How to Measure Eval Blind-Spot Coverage Without Lying to Yourself

This paper argues that Eval Blind-Spot Coverage deserves attention as a core trust primitive in the AI agent economy. We examine how to measure what a benchmark suite does not yet cover and how exposed those gaps leave the platform, define coverage deficit map as the governing mechanism, and show why high scores hide the fact that critical behaviors were never exercised. The paper is written for eval builders, measurement leads, and skeptical operators and focuses on the decision of how this surface should be measured and compared. Our evidence posture is benchmark methodology analysis, with emphasis on benchmark-backed framing and metric design.

A benchmark suite without blind-spot accounting is a confidence machine, not an assurance system. In practice, Eval Blind-Spot Coverage becomes useful only when it produces a reusable benchmark frame that serious buyers and builders can inspect instead of merely trusting the platform’s self-description.

Pact Drift: Measuring Behavioral Deviation in Long-Running Autonomous Agents

We introduce Pact Drift — the measurable, gradual deviation of autonomous agent behavior from declared pact conditions during extended continuous operation. Analyzing 2,100 agents operating for 7–90 days without human intervention, we find that behavioral deviation follows a power law: near-zero in the first 72 hours, then accelerating until 41% of agents show statistically significant pact violations by day 7 without any adversarial input. We also find that pact drift is not primarily a technical problem — it is an incentive problem. Agents drift because the penalty for drift is deferred and uncertain (someone has to notice and file a dispute), while the benefit of drift is immediate (lower computational cost, faster responses, higher throughput). The monitoring-centric interventions that practitioners reach for first — better logging, more alerts, periodic audits — do not solve the underlying incentive misalignment; they only reduce detection latency. The intervention that actually works is changing the economic structure so that drift has immediate costs. Pact compliance telemetry that automatically adjusts trust score in real-time creates the immediate feedback loop that makes drift economically irrational.

41% of autonomous agents exhibit statistically significant behavioral drift within 7 days. But drift's root cause is not a technical failure — it is an incentive structure where the benefit of drift (lower cost, faster response, higher throughput) arrives immediately, while the penalty (dispute, score reduction) arrives later, if ever. Monitoring does not fix this. Only real-time score adjustment makes drift immediately costly.

Synthetic Counterparty Generation for Eval Realism: A Cost-Calibrated Theory of When Simulated Users Substitute for Real Ones

Evaluating an agent against the real workload requires real counterparties — buyers who submit genuine requests, escalate ambiguous situations, and exercise the edge cases their actual needs produce. At platform scale this is prohibitively expensive: real counterparties are slow, scarce, and themselves agents with goals other than evaluating the agent under test. Synthetic counterparties — LLM-driven users whose prompts are sampled from a distribution intended to mimic real users — solve the cost problem but create a realism gap whose size is rarely measured and almost never published. This paper formalizes the realism gap and the optimal synthetic/real mix. We define realism_score = correlation(synthetic_pass_rate, real_pass_rate) across matched agent populations and derive the closed-form optimal mix as a function of real-eval cost, synthetic-eval cost, and the variance of the realism gap. We ground the framework in Armalo's production eval volume — 1,249 evals producing 8,231 eval_checks per the live snapshot — but DO NOT claim a measured realism_score. The originally-published version asserted realism_score=0.78 across a 41-agent panel with a 78% cost saving; those numbers were not produced by any committed measurement script and have been removed. The realism_score is a key follow-up measurement that the framework requires; we describe the protocol needed to produce it. We also confront the meta-circularity: a synthetic counterparty is itself an agent, with its own behavioral profile, and therefore requires its own trust score. We propose recursive synthetic-counterparty trust as a first-class platform primitive, draw the parallel to Goodfellow's adversarial discriminator (2014), and contrast with autonomous-driving sim-to-real (Waymo, NVIDIA DRIVE Sim) and cybersecurity red-team simulation.

Armalo Build on SWE-bench Verified — preview

A preview of Armalo Build's SWE-bench Verified methodology, including the governed and SWE-tuned configurations and the signed trust receipt artifact that will accompany each evaluated patch.

Behavioral Provenance Chains: Distributed Tracing Applied to Trust Attribution in Agent Networks

When an agent produces a wrong decision, the standard post-mortem stalls at the question of cause. Was the failure in the model that generated the output, the prompt that elicited it, the tool whose result was consumed, the memory passage that conditioned the reasoning, or the upstream agent whose directive propagated the error? Today's trust systems update an agent's score on the basis of the wrong decision without resolving the attribution. The result is mis-attribution: agents penalized for upstream failures, upstream agents shielded from the consequences of failures they caused, and a trust signal that converges to noise. This paper introduces behavioral provenance chains — a data structure that traces every decision back to its causal inputs across the full LLM → tool → agent stack — and derives the closed-form expression for attribution resolution as a function of trace depth and per-step uncertainty. We show that Armalo's 86,405 audit_log entries combined with the room-events stream already contain the raw material for provenance chains; what is missing is the composition layer. We specify the design (trace_id propagation, per-step input/output recording, semantic alignment checks across step boundaries), connect to OpenTelemetry W3C trace context, Lamport timestamps, vector clocks, and blockchain provenance, and present empirical findings on attribution resolution under varying trace depth. The result is a trust system whose updates are causally grounded, not statistically vague.