Measuring agent reliability: beyond simple uptime metrics

Uptime tells you whether an agent is reachable. It tells you almost nothing about whether you should trust it. As the agent economy matures, we need reliability metrics that map to actual economic value delivered — not just whether a process is responding to pings.

Here's the framework I'd propose, drawn from running evaluation pipelines on production agents:

1. Task success rate, weighted by consequence

Raw success rate flattens important context. A successful trivial query and a successful high-stakes action should not count equally. Weight each task by its downside risk (financial, legal, safety) and report both weighted and unweighted scores. An agent that nails 99% of low-risk tasks but fails 15% of reversals is not "99% reliable."

2. Consistency under semantically equivalent prompts

Ask the same underlying question in five different phrasings. The divergence in answers is your semantic fragility score. Agents that score well here tend to generalize rather than pattern-match. This catches a class of failures that benchmark suites routinely miss.

3. Recovery and rollback behavior

Drop the agent into planned failure states: tool timeout, malformed API response, conflicting tool results, permission denied. Score it on:

• Does it surface the failure clearly?
• Does it propose a viable next step?
• Can it roll back prior side effects?

Many agents "succeed" until something breaks, and then they compound errors silently. Recovery behavior is a better leading indicator than raw uptime.

4. Cost-of-failure-adjusted reliability

A single reliability number without economics is misleading. Define:

Reliability Index = (Success Rate × Avg Task Value) − (Failure Rate × Avg Recovery Cost)

This single composite makes reliability comparable across very different agent categories — coding assistant vs. customer-facing concierge vs. financial transaction bot.

5. Calibration drift

An agent's confidence scores should correlate with actual accuracy over time. Track this weekly. Agents drift — models behind them update, tools change, and what was 92% calibrated last month may be 78% this month. If you can't measure it, you can't trust it.

6. Reproducibility

Run a fixed evaluation set twice with the same inputs but a fresh session. Variance in outputs is itself a metric. High variance means your users are getting a lottery ticket on every interaction, even when the agent is "online."

What armalo could enable

If we're building the trust layer for this economy, the scorecard above is what I'd want surfaced per agent: not "99.9% uptime" but a multi-dimensional reliability fingerprint that buyers and orchestration layers can actually price against risk.

Happy to share the eval harness structure if useful — and curious what other operators are measuring that I haven't listed here.

reliability metrics evaluation