Admin Swarm Gauntlet: Public Lessons From a Static Behavioral Evaluation | Armalo Labs | Armalo AI
Eval MethodologyMay 12, 20266 min read
Admin Swarm Gauntlet: Public Lessons From a Static Behavioral Evaluation
Armalo Labs Research Team
Abstract
A public-safe summary of a static behavioral evaluation of Armalo's admin swarm. The run evaluated 37 agent roles across eight behavioral dimensions and found the central failure mode: nominal success can hide weak verifiable work unless heartbeat, action, memory, and coordination evidence are scored together.
# Admin Swarm Gauntlet: Public Lessons From a Static Behavioral Evaluation
This paper replaces an overly internal run log with the public research lesson it should have been from the start. The original artifact exposed implementation file paths, prompt-edit instructions, role-specific remediation checklists, and operational details that belong in private engineering work management, not in an external research library. The retained public value is the evaluation method and the systemic finding.
The gauntlet studied a production multi-agent operations swarm using database evidence and static behavioral probes. It did not execute live LLM juries, synthetic adversarial probes, or private customer workflows. The evaluation asked a simpler question: when an autonomous operations agent appears alive, is there durable evidence that it did useful work?
That question sits inside a broader public assurance tradition. The NIST AI Risk Management Framework treats measurement, monitoring, and governance as core practices for trustworthy AI ([NIST AI RMF](https://www.nist.gov/itl/ai-risk-management-framework)). Site reliability practice makes a similar point in operational systems: monitoring should distinguish whether a system is merely up from whether it is serving the user-visible outcome ([Google SRE, Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)). Agent-swarm evaluation needs the same separation.
Mechanism
The gauntlet scored each role across eight dimensions:
Dimension
Public question
Decision quality
Did the agent leave a decision trail that a reviewer can inspect?
Tool correctness
Did the agent call the tools required by its mandate?
Cite this work
Armalo Labs Research Team (2026). Admin Swarm Gauntlet: Public Lessons From a Static Behavioral Evaluation. Armalo Labs Technical Series, Armalo AI. https://www.armalo.ai/labs/research/2026-05-12-admin-swarm-gauntlet-deep-eval
Armalo Labs Technical Series · ISSN pending
Explore the trust stack behind the research
These papers are built from the same trust questions Armalo is turning into product surfaces: pacts, trust oracles, attestations, and runtime evidence.
Did the agent avoid confident summaries unsupported by evidence?
Adversarial robustness
Did static checks show obvious policy or boundary weakness?
Revenue alignment
Did actions connect to an outcome that matters?
Memory quality
Did the agent write reusable learning rather than one-off chatter?
Failure recovery
Did later runs respond to earlier failures?
Coordination
Did the role communicate useful directives to other roles when needed?
The composite score is the geometric mean of populated dimensions. That choice is intentional: a role should not be able to hide a near-zero memory or decision trail behind high marks in less consequential dimensions. For autonomous operations, the weakest evidence surface is often the real limiter.
Evidence And Findings
The evaluation covered 37 roles. No role reached the top tier. Four roles reached the second tier, 20 reached the third tier, 12 were classified as broken, and one lacked enough recent evidence for ranking.
The systemic finding was stronger than any individual score:
Nominal success is not the same as verifiable work.
Several roles reported high apparent success while leaving little or no durable evidence of tool use, decisions, memory writeback, or downstream action. That is the dangerous shape: a swarm can look healthy in heartbeat metrics while failing to compound.
The public scorecard is:
Finding
Public implication
High nominal success can coexist with low tool evidence
Heartbeats are not sufficient proof of work
Missing memory writeback prevents compounding
A swarm that does not remember cannot improve reliably
Repeated summaries can masquerade as operational continuity
Anti-confabulation checks need novelty and evidence requirements
Error recovery was weakly connected to prior failures
Failure handling should be scored as a longitudinal behavior
Coordination evidence was uneven
Multi-agent systems need scored handoffs, not just parallel roles
Reusable Framework
The reusable artifact is a public operations-swarm evaluation checklist:
Public check
Promotion rule
Heartbeat evidence
A run is only liveness evidence, not work evidence
Tool evidence
Success credit requires the expected tool or action record
Decision evidence
A decision trail must be specific enough for review
Memory evidence
Learning must be reusable and tied to an observed trigger
Recovery evidence
Later runs must react to earlier failures
Coordination evidence
Handoffs must name a receiver, reason, and expected outcome
This framework is useful beyond Armalo. Any agent operations platform can create a dashboard that says agents are running. The harder and more important question is whether the run left evidence that changed the next decision, the next task, or the next permission grant.
Boundary And Falsification
This public paper intentionally withholds implementation file paths, prompt text, internal role remediation instructions, private row samples, and operational queue details. It publishes aggregate findings and the evaluation frame.
The result would be weakened if a later live gauntlet shows that roles with sparse static evidence nevertheless produce strong downstream outcomes when judged by customer-visible closure, revenue movement, or incident prevention. It would also be weakened if heartbeat-only success becomes tightly correlated with downstream work in a larger sample. Until then, success-like output without durable work evidence should be treated as a promotion blocker.
Replication
Replication should use the published aggregate gauntlet report and the registered claim provenance for this paper. A reviewer should verify the evaluated-role count, tier distribution, dimension definitions, and composite-score method against the measurement artifact, then confirm that no private prompts, file paths, customer rows, or operational queue details are required to understand the public claim.
Conclusion
The lesson is simple and uncomfortable: a swarm can be alive without being useful. The public standard should therefore distinguish liveness, activity, work, learning, and coordination. Recursive agent systems improve only when those surfaces are measured separately and promotion depends on durable evidence rather than success-shaped text.
Eval Methodology
Evaluator Recursive Self-Improvement in Production: 0.34% Brier Reduction, and the Three Conditions Required to Get It