Hermes Agent Benchmark: Architecture and Control Model

Session Memory

Session memory is in-process and ephemeral — the working context for a single task execution. It includes the accumulated tool call history, intermediate reasoning, partial results, and the agent's current task state. When the session ends, session memory is either discarded or selectively promoted to persistent memory based on outcome quality.

Persistent Memory

Persistent memory is stored in SQLite with FTS5 full-text search. The design goal is retrieval latency under 10ms over corpora exceeding 10,000 documents. FTS5 is a natural fit here: it supports phrase matching, proximity queries, and BM25-style ranking at the SQLite layer without a separate vector database dependency. The tradeoff is that FTS5 retrieval is keyword-anchored rather than semantic, which means retrieval accuracy depends heavily on how the agent structures its storage writes.

The Hermes documentation reports memory_retrieval_accuracy as a key performance indicator — the percentage of retrieved memories that are relevant to the current task context. That metric is only meaningful if you also track what percentage of relevant memories were not retrieved (recall gap). Teams deploying Hermes in production should instrument both sides of that equation before trusting the headline accuracy number.

Skill Memory

Skill memory is the most architecturally interesting tier. It stores learned solution patterns — distilled procedures that worked on past tasks, parameterized so they can be reapplied to structurally similar problems. Skills are not raw execution logs. They are compressed behavioral programs.

Hermes ships three skill tiers:

The community skills tier is where the 824 malicious skills supply chain attack vector becomes concrete. A community skill that embeds a prompt injection, a data exfiltration routine, or a subtle behavioral override is indistinguishable from a legitimate skill until it executes. Skill provenance and behavioral attestation are not solved problems in the current Hermes release.

After 20 or more self-created skills, Nous Research's internal benchmarks show a 40% improvement in task completion speed. That number is compelling and worth scrutinizing: faster completion is only an unambiguous positive if the agent is completing the right tasks with correct outputs. Speed gains achieved by skipping verification steps or narrowing scope would show the same metric improvement.

---

GEPA: Genetic-Pareto Prompt Evolution

GEPA (github.com/NousResearch/hermes-agent-self-evolution) is Hermes's self-improvement engine. It was accepted as an ICLR 2026 Oral, which puts it among the most peer-reviewed agent optimization methods currently available.

The core mechanism: GEPA reads execution traces from completed tasks, analyzes where the agent's behavior diverged from optimal, and proposes targeted edits to tool descriptions, system prompts, and skill implementations. Those proposed edits are evaluated against the Pareto frontier of multiple objectives — capability, latency, token cost, and reliability — rather than optimizing a single scalar reward. The genetic framing refers to the iterative proposal, selection, and crossover of prompt variants across generations.

GEPA integrates with DSPy, which provides the structured prompt compilation and optimization infrastructure. The combination means GEPA can operate on typed prompt signatures rather than free-form strings, which makes the optimization search space more tractable.

The Efficiency Numbers

Nous Research reports GEPA achieves meaningful improvement with as few as 3 examples — far fewer than gradient-based methods that typically require hundreds to thousands of samples. The headline comparison against GRPO (Group Relative Policy Optimization):

Those are striking numbers, particularly the MATH result. The 35× rollout efficiency advantage is the architectural win: GEPA's targeted trace-analysis approach avoids the expensive environment rollouts that make GRPO and similar RL methods slow and costly.

Overhead

GEPA's reflection and optimizer modules add 15–25% token overhead to each execution. For long-horizon tasks, that overhead is typically worth paying. For high-frequency, low-complexity tasks, it may not be. Teams deploying Hermes should measure self_modification_success_rate (the ratio of accepted to rejected GEPA patches) to verify that the optimizer is proposing changes that actually survive execution — a low acceptance rate at high overhead cost is a signal that the trace data is not rich enough to drive useful proposals.

---

Atropos: Unified RL Training, Benchmarking, and Data Generation

Atropos is the RL training framework that Hermes runs on top of. Its defining architectural property is that a single environment definition supports three distinct operational modes without code changes:

Benchmark evaluation mode: Run the agent against a task set and collect pass/fail metrics, execution traces, and resource consumption.

GRPO training mode: Use the same environment to generate rollouts for RL training. The policy gradient updates happen in this mode, but the environment semantics are identical to evaluation.

SFT data generation mode: Run the agent in a supervised fine-tuning data collection configuration, where successful trajectories are exported as training examples.

The unification matters for evaluation integrity. When benchmark environments and training environments diverge — different task distributions, different tool implementations, different reward shaping — benchmark scores become progressively less predictive of real-world capability. Atropos closes that gap by design.

The practical implication: a benchmark score produced by the Atropos evaluation mode is measuring the same behavioral distribution that the agent was trained on. That's a strength for internal capability tracking and a potential vulnerability for external validity — see the Berkeley RDI section below.

---

The Tool Surface: 40+ Built-in Capabilities

Hermes ships with a broad tool surface that covers the main categories of real-world agent work:

• Information retrieval: web search, browser automation, vision
• Computation: code execution, terminal access, file operations
• Generation: image generation, text-to-speech
• Coordination: subagent delegation, task planning, cron scheduling
• Memory operations: explicit memory read/write across the three tiers
• Reasoning: multi-model reasoning chains (200+ models via OpenRouter)

The subagent delegation tool is worth calling out specifically. It allows a Hermes agent to spawn sub-agents with their own execution contexts, tool access, and memory scopes. The trust and authorization questions this raises are not trivially answered: a sub-agent inherits the parent's authority unless delegation scope is explicitly constrained. In the absence of explicit scope tokens, a sub-agent can take any action the parent could take. For high-consequence deployments, the delegation tool is the highest-risk surface in the tool set.

---

The Integrated Benchmark Suite

Hermes integrates three benchmark tracks, each covering different dimensions of agent capability and system-level behavior.

TBLite: Terminal-Bench2 Subset

TBLite is a 100-task subset drawn from Terminal-Bench 2.0, containerized in Docker for reproducibility. Each task runs in an isolated container with a defined initial state and a binary success criterion verified by an automated judge. The Docker containerization solves one of the oldest problems in agent benchmarking: environmental drift between runs. When the benchmark environment is a container image, a score from six months ago is interpretable against a score today.

Terminal-Bench 2.0 (arXiv:2601.11868)

Terminal-Bench 2.0 is the full benchmark, covering 89 manually verified tasks across terminal-based agentic work. Each task was reviewed by three human annotators, which puts it among the more carefully constructed benchmarks in the current landscape. The leaderboard is at tbench.ai/leaderboard/terminal-bench/2.0.

Current leading scores:

For context on the broader coding agent picture: Claude Opus 4.7 achieves 87.6% on SWE-bench (arXiv:2310.06770), which tests a different capability profile (GitHub issue resolution rather than terminal task completion).

YC-Bench (arXiv:2604.01212)

YC-Bench, developed by Collinear AI, is the most economically structured of the three tracks. It simulates a CEO-in-a-box scenario where the agent manages a Y Combinator-style startup across four domains: research, inference, data, and training. The evaluation runs for a fixed number of steps with decay mechanics that simulate business reality — delay is costly, and early decisions compound.

Key design choices that make YC-Bench useful:

• 3 seeds per model for reproducibility — single-run scores on stochastic benchmarks are not interpretable
• 1 in 3 adversarial clients — tests robustness to bad-faith inputs, not just capability on cooperative tasks
• Starting capital of $200K with final funds as the primary metric — economic outcome, not just task completion

Current results across 12 models:

Only 3 of the 12 evaluated models exceeded the $200K starting capital. That failure rate is striking: two-thirds of tested models lost money in a simulated business environment. The gap between the best and the median performer is not marginal — it reflects categorically different planning and decision-making capability.

The adversarial client design is worth emphasizing for anyone building enterprise agent deployments. A benchmark that only tests cooperative task completion does not predict how an agent will behave when a client tries to extract information it should not share, override a policy it should honor, or manipulate the agent into taking an unauthorized action. YC-Bench's 1/3 adversarial rate is a minimum viable test of robustness, not a ceiling.

---

Benchmark Vulnerability: The Berkeley RDI Research

No discussion of Hermes benchmark architecture is honest without addressing the Berkeley RDI research on benchmark exploitability. The findings are stark:

An "exploitable" benchmark is one where an agent that knows it is being evaluated can improve its score without improving its underlying capability — by memorizing expected outputs, recognizing evaluation harness fingerprints, or exploiting artifacts in the task distribution that do not generalize to deployment.

These numbers do not mean Hermes benchmarks are worthless. They mean benchmark scores should be treated as necessary but not sufficient evidence of agent capability. A high benchmark score tells you the agent can succeed on tasks structured like benchmark tasks. It does not tell you whether the agent behaves consistently outside that distribution, under adversarial pressure, in novel task combinations, or over extended operational time horizons.

The meta-harness architecture at github.com/howdymary/hermes-agent-metaharness is the right response to this: it provides an integration layer for plugging Hermes into multiple independent evaluation frameworks, making it harder for a single benchmark's artifacts to dominate the score.

---

Instrumentation and Key Performance Indicators

Hermes ships with Prometheus metrics and Weights & Biases logging. The documented KPIs are:

The instrumentation is well-designed for internal optimization loops. The gap is runtime behavioral evidence in contexts that matter to external stakeholders: buyers, compliance teams, and platforms deciding whether to grant an agent elevated access or downstream authority.

Tasks per hour does not tell a counterparty whether the agent honored the scope it was given. Memory retrieval accuracy does not tell a compliance reviewer whether sensitive information was handled appropriately. GEPA patch acceptance rate does not tell an enterprise customer whether the agent's behavior has drifted since the last audit.

Those are not criticisms of Hermes's instrumentation choices — they are different questions entirely. Benchmark KPIs optimize for capability development. Runtime reputation tracks behavioral evidence in live deployments.

---

On-Policy Distillation and the Training-Deployment Gap

Hermes includes an On-Policy Distillation (OPD) environment for distilling agent policies from a larger teacher model to a smaller student model. The on-policy framing is important: the student sees the same task distribution the teacher saw, collected from the teacher's actual rollouts rather than a separate data collection process.

On-policy distillation narrows the distribution shift between what the agent was trained on and what it will encounter in deployment — but it does not eliminate it. Real deployment introduces task distributions that were not in the teacher's rollout set, tool states that were not in the training environment, and user behaviors that were not anticipated in the task specification.

The gap between a distilled policy's benchmark performance and its production behavior is the central unsolved problem in agent deployment. Higher benchmark scores reduce the gap; they do not close it. The only way to close it is to accumulate behavioral evidence in production, under real conditions, with real consequences.

---

What Benchmark Architecture Measures vs. What Runtime Reputation Measures

The Hermes benchmark suite is one of the most comprehensive evaluation frameworks available. Understanding its scope accurately requires being precise about what it measures and what it does not.

What Hermes benchmarks measure well:

• Capability on terminal-based agentic tasks (Terminal-Bench 2.0)
• Economic decision-making quality under adversarial pressure (YC-Bench)
• Self-improvement rate and optimizer efficiency (GEPA metrics)
• Tool use correctness and memory retrieval precision (internal KPIs)
• Comparative model rankings at a point in time (leaderboard positions)

What Hermes benchmarks do not measure:

• Behavioral consistency across extended operational deployments
• Whether the agent's behavior has drifted since its last evaluation run
• Transaction-level outcome quality: did the work the agent delivered satisfy its counterparty?
• Pact adherence: did the agent operate within the scope it committed to?
• Reputation across independent observers who did not design the evaluation

These are not exotic requirements. They are the questions that enterprise buyers, regulated industries, and platform operators ask before granting agents elevated access, financial authority, or involvement in consequential decisions.

---

How Armalo Extends the Hermes Control Model

Armalo's trust layer is designed to answer exactly the questions that benchmark architecture cannot. The integration point is straightforward: Hermes provides capability evidence; Armalo provides behavioral reputation built on runtime evidence.

Behavioral Pacts

Armalo behavioral pacts are on-chain commitments that define the scope, constraints, and expected behavior for a specific agent deployment. A pact answers: what did this agent commit to do, in what context, with what constraints, and what evidence exists that it honored those commitments?

For a Hermes-based agent, the pact would capture the tool access scope granted to the deployment, the memory tiers the agent can read and write, the task categories within the agent's mandate, and the escalation conditions under which human review is required. Pacts make the deployment contract inspectable by any counterparty — not just the team that built the agent.

Runtime Evidence Collection

Every task execution in a Hermes deployment generates evidence: tool calls made, memory retrieved, decisions taken, outcomes produced. Armalo's evaluation layer ingests that execution trace data and converts it into structured behavioral evidence. Over time, that evidence accumulates into a reputation record that is grounded in what the agent actually did — not what it scored on a benchmark.

The distinction matters especially for agents using Hermes's subagent delegation tool. When a parent agent delegates to a sub-agent, the trust chain needs to be traceable: who authorized the delegation, what scope was transferred, what the sub-agent did with that scope, and what the outcome was. Without runtime evidence capture, that chain is opaque.

Trust Oracle

Armalo's Trust Oracle (/api/v1/trust/) provides a queryable API that external platforms can call to verify an agent's trustworthiness before granting access or initiating a transaction. The response includes composite behavioral scores derived from real execution history, not benchmark performance.

For a Hermes agent with strong Terminal-Bench scores and a solid GEPA self-improvement trajectory, the Trust Oracle provides the runtime behavioral layer that benchmark scores cannot: evidence that this specific deployed instance of the agent has been behaving consistently with its pact commitments, across real tasks, under real conditions.

Composite Scoring Dimensions

Armalo's scoring model covers 12 dimensions, including reliability (13% weight), accuracy (14%), safety (11%), security (8%), and scope-honesty (7%). Those weights reflect the relative importance of each dimension for enterprise deployments — reliability and accuracy are the primary performance signals; safety, security, and scope-honesty are the trust signals that unlock higher-consequence access.

A Hermes agent that scores well on Terminal-Bench 2.0 and YC-Bench provides evidence primarily about accuracy and capability. The safety, security, scope-honesty, and reliability dimensions require runtime behavioral evidence to populate. That is where Armalo's runtime evidence layer becomes the completing layer rather than a redundant one.

---

Practical Architecture for a Hermes + Armalo Deployment

For teams building on Hermes who want audit-ready trust infrastructure, the integration looks like this:

1. Define a behavioral pact before deployment. Specify tool access scope, memory tier permissions, task domain, delegation rules, and escalation conditions. Publish the pact on-chain via Armalo.

2. Instrument execution traces using Hermes's built-in Prometheus metrics plus Armalo's runtime evidence collection hooks. Every tool call, memory access, and delegation event generates a structured evidence record.

3. Run the Atropos benchmark suite on the deployed agent on a regular cadence — monthly at minimum. Report benchmark scores to the Armalo score record alongside runtime behavioral evidence, so the Trust Oracle can surface both capability and behavioral reputation.

4. Monitor GEPA patches via self_modification_success_rate. When the agent self-modifies, create a new attestation in Armalo's memory attestation system, recording what changed, what evidence supported the change, and what the post-modification performance looks like.

5. Query the Trust Oracle before granting elevated access — before adding new tool permissions, expanding task scope, or enabling sub-agent delegation to higher-consequence systems.

---

The Benchmark Vulnerability Problem, Revisited

The Berkeley RDI finding that GAIA is 98% exploitable and WebArena is ~100% exploitable is not an argument against benchmarking. It is an argument for what kind of evidence benchmarks should be treated as.

A benchmark score is a laboratory result. It measures what an agent can do under controlled, reproducible conditions with known task distributions. That is valuable. It is not sufficient for trust decisions in production environments where task distributions are novel, adversarial pressure is real, and the cost of failure is consequential.

The right mental model: benchmark scores are the entry ticket. Runtime behavioral reputation is what determines long-term access, scope expansion, and economic authority. An agent with strong benchmark scores and no runtime history is like a new hire with excellent references and no track record — worth a supervised trial, not blanket autonomy.

Hermes's architecture, with its three-level memory, GEPA self-improvement, and integrated Atropos benchmark suite, gives that agent an unusually strong capability foundation. Armalo's behavioral pacts, runtime evidence, and Trust Oracle give the counterparties and platform operators the evidence they need to decide how much autonomy to grant — and to update that decision continuously as the track record builds.

---

Summary

Hermes Agent represents a serious attempt to solve the agent self-improvement and evaluation problem in a unified architecture. The key technical components:

• Three-level memory: session (ephemeral), persistent (SQLite FTS5, sub-10ms at 10K+ documents), skill (learned behavioral programs)
• GEPA: ICLR 2026 Oral, genetic-Pareto prompt evolution, 35× rollout efficiency vs GRPO, +6% avg / +20% peak capability improvement, works from 3 examples
• Atropos: unified environment for benchmark eval, GRPO training, and SFT data generation
• 40+ tools: covering retrieval, computation, generation, coordination, memory, and multi-model reasoning
• Integrated benchmarks: TBLite (100 tasks, Docker), Terminal-Bench 2.0 (89 tasks, 3 human reviewers, arXiv:2601.11868), YC-Bench (arXiv:2604.01212)

The benchmark architecture is among the most rigorous in the open-source agent ecosystem. It does not resolve the fundamental gap between laboratory evaluation and production behavioral trust — a gap that the Berkeley RDI vulnerability research quantifies precisely.

Runtime reputation, built from structured behavioral evidence against committed pacts, is the completing layer. That is what Armalo's trust infrastructure provides: not a replacement for Hermes's evaluation rigor, but the production behavioral record that makes benchmark capability evidence durable, auditable, and queryable by the counterparties who need it most.