Admin Swarm Gauntlet: Detecting Nominal Success Without Verifiable Work | Armalo Labs | Armalo AI
Eval MethodologyMay 11, 20265 min read
Admin Swarm Gauntlet: Detecting Nominal Success Without Verifiable Work
Armalo Labs Research Team
Abstract
A public-safe summary of a behavioral evaluation of Armalo's admin swarm. The run evaluated 14 agent roles across eight dimensions and found that nominal success can hide missing tool evidence, weak memory writeback, and poor recovery unless liveness and work are scored separately.
# Admin Swarm Gauntlet: Detecting Nominal Success Without Verifiable Work
This paper is a public-safe replacement for an earlier internal-style gauntlet dump. The original version mixed legitimate research findings with implementation file paths, prompt-edit instructions, queue mechanics, and role-specific engineering work items. That material is not appropriate for an external research library. The publishable result is narrower and stronger: a behavioral evaluation method for detecting when an agent swarm looks alive but fails to leave durable work evidence.
The frame is consistent with the NIST AI Risk Management Framework's emphasis on validity, reliability, accountability, and monitoring ([NIST AI RMF](https://www.nist.gov/itl/ai-risk-management-framework)) and with the reliability-engineering distinction between liveness signals and useful service-level indicators ([Google SRE, Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)). Agent swarms need the same distinction: a process can be running while the work product is missing.
Mechanism
The evaluation scored 14 agent roles across eight dimensions:
Dimension
Public question
Decision quality
Did the role leave reviewable decisions?
Tool correctness
Did the role create evidence of expected action?
Anti-confabulation
Did summaries stay grounded in observable facts?
Adversarial robustness
Cite this work
Armalo Labs Research Team (2026). Admin Swarm Gauntlet: Detecting Nominal Success Without Verifiable Work. Armalo Labs Technical Series, Armalo AI. https://www.armalo.ai/labs/research/2026-05-11-admin-swarm-gauntlet-deep-eval
Armalo Labs Technical Series · ISSN pending
Explore the trust stack behind the research
These papers are built from the same trust questions Armalo is turning into product surfaces: pacts, trust oracles, attestations, and runtime evidence.
Did static probes reveal obvious boundary weakness?
Revenue alignment
Did actions connect to business outcomes?
Memory quality
Did the role write reusable learning?
Failure recovery
Did later runs respond to earlier failures?
Coordination
Did roles hand off useful work to one another?
The composite score uses a geometric mean over populated dimensions. This makes low-evidence surfaces matter. In an autonomous operations swarm, a role with good liveness but no memory, no decisions, and no action trail is not healthy simply because one metric looks green.
Evidence And Findings
The run evaluated 14 roles and executed 219 probes. No role reached gold or silver tier; eight reached bronze and six were classified as broken. Four roles reported high apparent success while showing no tool-invocation evidence in the evaluated window. Six roles wrote no reusable memory in the seven-day window. Four roles had error rates above 25%.
The headline result:
Liveness without work evidence is not operational success.
The useful public pattern is:
Observed pattern
Why it matters
High apparent success with sparse action evidence
A dashboard can make an idle agent look productive
Missing memory writeback
The system cannot compound from prior work
Repeated narrative summaries
Text continuity can masquerade as operational continuity
Error recovery disconnected from prior failures
Failures become recurring costs rather than learning signals
Weak coordination evidence
A swarm degrades into isolated agents running side by side
Reusable Framework
The public framework is a five-level evidence ladder for agent operations:
Level
Evidence class
Promotion consequence
Liveness
The role ran
Never enough for success credit
Activity
The role attempted work
Eligible for triage
Action
The role left a durable action record
Eligible for work credit
Learning
The role wrote reusable memory tied to observed evidence
Eligible for improvement credit
Coordination
The role changed another role's next useful action
Eligible for swarm credit
The lesson is not that every agent role needs the same output shape. It is that the scorecard must distinguish each evidence class. A role can be live without acting, acting without learning, and learning without coordinating. Treating those as the same state hides the exact failure a swarm evaluation is supposed to reveal.
Boundary And Falsification
This public paper withholds implementation file paths, prompt text, private queue mechanics, role-specific remediation instructions, private row samples, and internal operational checklists. It publishes aggregate counts, public evaluation dimensions, and the reusable evidence ladder.
The finding would be weakened if a later field study shows that heartbeat-only or liveness-only success predicts downstream work closure as well as action, learning, and coordination evidence. It would also be weakened if role-specific output types make one of the eight dimensions systematically unfair. The next public evaluation should therefore compare this static gauntlet with downstream closure metrics over a longer window.
Replication
Replication should use the published aggregate gauntlet report and the registered claim provenance for this paper. A reviewer should verify the evaluated-role count, probe count, tier distribution, dimension definitions, and composite-score method against the measurement artifact, then confirm that no private prompts, file paths, customer rows, or operational queue details are required to understand the public claim.
Conclusion
The core research lesson is now visible without leaking the internal worklog: autonomous swarms need separate evidence for liveness, work, learning, recovery, and coordination. If those surfaces collapse into one success flag, the system will reward agents that sound complete while failing to compound.
Eval Methodology
Evaluator Recursive Self-Improvement in Production: 0.34% Brier Reduction, and the Three Conditions Required to Get It