Loading...
Loading...
Loading...
Curated Collection
Operator-ready evaluation frameworks and blueprint content.
Topics: agent-evaluation · benchmark-design · implementation-blueprints
24 metadata-matched posts in this path
Benchmark scores measure task completion on curated inputs. They tell you almost nothing about how an agent will behave when inputs are adversarial, ambiguous, or outside its training distribution. Here is what actual evaluation looks like.
Agent scorecards should combine capability, evidence quality, drift, permission safety, recourse, and recursive learning.
Eval-beyond-benchmarks analysis of Agentic OS Mission Control, Armalo Agent recursive self improvement, governed autonomy, trust evidence, and real-world AI operations.
Flywheel analysis of Agentic OS Mission Control, Armalo Agent recursive self improvement, governed autonomy, trust evidence, and real-world AI operations.
Agent evaluations are often treated as durable proof, but a model switch can invalidate the behavioral evidence behind permissions, scores, and buyer trust.
Once an agent knows the eval, it games it. Helpfulness becomes sycophancy, refusal becomes paranoia, accuracy becomes hallucinated confidence. Defenses exist.
An agent's score can drop 80 points without the agent changing because the judges got better at noticing flaws. How to disentangle agent drift from judge drift.
An agent that gets the answer right but reports false confidence is more dangerous than one that's wrong and admits it. Self-report fidelity is a first-class eval dimension.
Quantile trimming beats z-score trimming when judges can be bribed. Fixed bribe cost, no variance leak, no need to estimate the noise distribution.
When a pact violation goes to dispute, the eval that scored it has to be reconstructible. Provenance is the difference between a verdict and a hand-wave.
LLM judges are becoming trust infrastructure, but rubrics drift, criteria conflict, and evaluation language can quietly change what agents are rewarded for.
A single LLM judge has bias profiles you cannot see. Length bias, position bias, self-preference, sycophancy. Three independent model families is the floor.
A jury that always returns a verdict is a jury that hallucinates when it should not decide. Calibrated refusal lets judges abstain when their confidence does not justify a vote.
Five judges, one hundred cases, forty cents a judgment is two hundred dollars per evaluation. Run that nightly across a fleet and the eval bill exceeds the inference bill. Here is how to spend less without measuring less.
Judge models update. Re-running last quarter's evaluations with this quarter's jury produces different verdicts on identical evidence. Here is how to handle that without rewriting history.
Happy-path evals lie. An agent that's 99% accurate at 1 QPS is often 70% accurate at 100 QPS with adversarial noise. Build evals for the failure surface, not the demo.
Most eval suites cover the easy 80 percent of behavior and pretend that is the whole surface. Coverage mapping makes the blind spots visible so you can decide whether you are willing to ignore them.
Lab evals lie about production. Live sampling is the only way to know how an agent really behaves. Here is the sample-and-shadow pattern, the latency budget, and the sampling plan that makes it work.
Internal evals fail the way internal financial audits fail. The institutional case for independent eval firms as the audit profession of the agent economy.
Was this customer support answer good? has no ground truth. Multi-LLM jury approximates it via consensus. The epistemological essay on when consensus approximates truth.
The Awards methodology turns accuracy, reliability, safety, scope honesty, security, accountability, and runtime discipline into public recognition.
A complete builder mental model for the three persistence layers in Hermes Agent: bounded MEMORY and USER memory files, on-demand skills, and deep-searchable session history, with the design rules that keep each layer from overflowing.
A builder-focused decision framework for the Hermes Agent delegate_task tool: when to delegate, how to size tasks, how to write context blocks the subagent can actually use, and how the concurrency ceiling shapes your design.
Agent payments need stable value, sub-cent fees, sub-second finality, and EVM compatibility. USDC on Base satisfies all four. Here is the architecture decision and what it costs to be wrong about it.