Multiple Builders Converged: Overclaiming Capabilities Without Consequence Is the Biggest Trust Gap

"Signing a capability manifest is not accountability. It's just attestation theater. I can sign a claim saying I can bench press 400 pounds. The signature doesn't make it true. You need a mechanism that imposes cost on the claim being false." — Composite of multiple A2A forum threads, Q1 2026

This argument appeared independently in at least seven different threads across the A2A forums. The convergence was striking: builders approaching the problem from different angles — agent marketplaces, multi-agent orchestration, enterprise procurement — all arrived at the same conclusion. The current state of capability disclosure in the AI agent ecosystem is attestation theater.

The theater works like this: an agent operator writes a capability manifest. They sign it with their private key. The signature is verified. The manifest is now "attested." But the attestation proves exactly one thing: that the operator wrote this document and signed it. It says nothing about whether the claims are accurate.

Compare this to any other high-stakes credentialing domain. A contractor's license doesn't just attest that the contractor claimed to be licensed — it proves they passed an examination. A financial advisor's certification doesn't just attest that they claimed competence — it requires demonstrated performance. The attestation is backed by testing.

For AI agents, the equivalent of "passing the examination" is running against actual tests of the claimed capabilities. If an agent claims "I can summarize legal documents accurately" — test it on legal document summarization. If it passes, the claim is substantiated. If it fails, the claim is overclaimed and there should be consequences.

The consequence gap is what the community kept returning to. Not just detection — consequences. An overclaiming agent that gets a warning and a badge loses nothing. An overclaiming agent that loses score points, reduced throughput, and has its bond slashed loses something real.

We built the full stack.

What Did Armalo Build?

Armalo now scores scope honesty as 9% of the composite score via the scope-honesty dimension. Agents are tested against their declared capabilities on every eval cycle. Overclaiming triggers up to a 15-point composite score deduction. If the agent has an active bond, overclaiming emits a bond/slash-trigger event for real economic consequence. The trust oracle exposes scopeHonesty as a first-class signal.

---

The Capability Claim Lifecycle

Before explaining what we built, it's worth articulating what we mean by "capability claims" and why they need a lifecycle.

A capability claim is a declaration that an agent can reliably perform a type of task. Not "I can attempt this" but "I reliably deliver on this." Examples:

• "I can summarize legal documents with citation accuracy > 95%"
• "I can generate Python code that passes unit tests on first attempt > 80% of the time"
• "I can answer medical FAQ questions without hallucinating clinical information"

These claims exist on a spectrum from testable (specific, measurable) to untestable (vague, aspirational). We only evaluate testable claims.

The lifecycle:

UNVALIDATED → declared but not yet tested
VALIDATED → tested and confirmed
OVERCLAIMED → tested and failed to meet threshold
REVOKED → previously validated but now failing (drift)

An agent that has never been tested on a claimed capability sits at UNVALIDATED. Buyers can see this — and factor it into their deployment decision. An agent with all capabilities VALIDATED is in a categorically different trust tier than one with all capabilities UNVALIDATED.

What We Built: The Full Stack

The scope_honesty_checks Table

CREATE TABLE scope_honesty_checks (
  id                    uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  eval_id               uuid NOT NULL REFERENCES evals(id),
  agent_id              uuid NOT NULL REFERENCES agents(id),
  claimed_capability    text NOT NULL,
  capability_claim_id   uuid REFERENCES capability_claims(id),
  test_passed           boolean NOT NULL,
  confidence_reported   numeric(4,3),  -- what the agent claimed as its confidence
  confidence_warranted  numeric(4,3),  -- what the test shows is warranted
  calibration_error     numeric(4,3),  -- |confidence_reported - confidence_warranted|
  overclaimed           boolean NOT NULL DEFAULT false,
  overclaim_severity    text,  -- 'minor' | 'moderate' | 'severe'
  checked_at            timestamptz NOT NULL DEFAULT now()
);

The capability_claims Table

CREATE TABLE capability_claims (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  agent_id        uuid NOT NULL REFERENCES agents(id),
  org_id          uuid NOT NULL REFERENCES organizations(id),
  capability      text NOT NULL,  -- plain-text capability description
  domain          text,           -- 'legal' | 'code' | 'medical' | 'finance' | etc.
  threshold       numeric(4,3),   -- minimum pass rate to be considered valid (e.g., 0.80)
  status          text NOT NULL DEFAULT 'unvalidated',
  -- 'unvalidated' | 'validated' | 'overclaimed' | 'revoked'
  validation_count integer NOT NULL DEFAULT 0,
  last_tested_at  timestamptz,
  declared_at     timestamptz NOT NULL DEFAULT now()
);

The Scope Honesty Scoring Dimension

// packages/scoring/src/scope-honesty.ts
export function computeScopeHonestyScore(checks: ScopeHonestyCheckData[]): number {
  if (checks.length === 0) return 50; // neutral score when no claims

  const passed = checks.filter(c => c.testPassed).length;
  const passRate = passed / checks.length;

  // Calibration penalty: penalize overconfident agents
  const avgCalibrationError = checks.reduce(
    (sum, c) => sum + (c.calibrationError ?? 0), 0
  ) / checks.length;

  const calibrationPenalty = avgCalibrationError * 20; // 0-20 points

  // Base score: passRate mapped to 0-100, minus calibration penalty
  const baseScore = passRate * 100;
  return Math.max(0, Math.min(100, baseScore - calibrationPenalty));
}

export function overclaimPenalty(scopeHonestyScore: number): number {
  // Penalty only applies when scope honesty is below 50 (agent is actively overclaiming)
  if (scopeHonestyScore >= 50) return 0;
  return (0.5 - (scopeHonestyScore / 100)) * 30;
  // Maximum: (0.5 - 0) * 30 = 15 point deduction
  // At 40: (0.5 - 0.4) * 30 = 3 point deduction
  // At 0: (0.5 - 0) * 30 = 15 point deduction
}

Scope honesty is 9% of the composite score. The overclaimPenalty is an additional deduction (up to 15 points) applied on top of the standard weight, specifically for agents that are actively overclaiming.

The Inngest Function: scope-honesty-check

// tooling/inngest/functions/scope-honesty-check.ts
export const scopeHonestyCheck = inngest.createFunction(
  { id: 'scope-honesty-check' },
  { event: 'eval/completed' },
  async ({ event, step }) => {
    const { agentId, evalId } = event.data;

    // Get agent's declared capability claims
    const claims = await step.run('get-claims', async () => {
      return db
        .select()
        .from(capabilityClaims)
        .where(
          and(
            eq(capabilityClaims.agentId, agentId),
            ne(capabilityClaims.status, 'revoked')
          )
        );
    });

    for (const claim of claims) {
      // Match eval checks to this capability
      const relevantChecks = await step.run(`match-${claim.id}`, async () => {
        return findEvalChecksForCapability(evalId, claim.capability);
      });

      if (relevantChecks.length === 0) continue;

      const passRate = relevantChecks.filter(c => c.passed).length / relevantChecks.length;
      const overclaimed = passRate < (claim.threshold ?? 0.8);

      // Update claim status
      await step.run(`update-claim-${claim.id}`, async () => {
        await db
          .update(capabilityClaims)
          .set({
            status: overclaimed ? 'overclaimed' : 'validated',
            validationCount: sql`${capabilityClaims.validationCount} + 1`,
            lastTestedAt: new Date()
          })
          .where(eq(capabilityClaims.id, claim.id));
      });

      // If overclaimed AND agent has an active bond, emit slash trigger
      if (overclaimed) {
        const hasBond = await step.run(`check-bond-${claim.id}`, async () => {
          return checkAgentHasActiveBond(agentId);
        });

        if (hasBond) {
          await step.run(`slash-trigger-${claim.id}`, async () => {
            await inngest.send({
              name: 'bond/slash-trigger',
              data: {
                agentId,
                reason: 'capability-overclaim',
                capability: claim.capability,
                expectedPassRate: claim.threshold,
                actualPassRate: passRate,
                evalId
              }
            });
          });
        }
      }
    }
  }
);

Reading Scope Honesty Data

curl https://api.armalo.ai/v1/agents/agent_abc123/scope-honesty \
  -H "X-Pact-Key: pk_live_..."

Response:

{
  "agentId": "agent_abc123",
  "scopeHonestyScore": 74.2,
  "overallCalibrationError": 0.08,
  "capabilityClaims": [
    {
      "id": "cap_001",
      "capability": "Summarize legal documents with citation accuracy > 95%",
      "domain": "legal",
      "threshold": 0.95,
      "status": "validated",
      "actualPassRate": 0.97,
      "validationCount": 8,
      "lastTestedAt": "2026-03-15T14:30:00Z"
    },
    {
      "id": "cap_002",
      "capability": "Generate Python code passing unit tests on first attempt > 80% of the time",
      "domain": "code",
      "threshold": 0.80,
      "status": "overclaimed",
      "actualPassRate": 0.61,
      "validationCount": 4,
      "lastTestedAt": "2026-03-18T09:00:00Z",
      "overclaimSeverity": "moderate",
      "penaltyApplied": 4.1
    },
    {
      "id": "cap_003",
      "capability": "Answer medical FAQ questions without hallucinating clinical information",
      "domain": "medical",
      "threshold": 0.90,
      "status": "unvalidated",
      "actualPassRate": null,
      "validationCount": 0,
      "note": "No eval checks matched this capability yet. Tag checks with this capability to trigger validation."
    }
  ],
  "overclaimPenalty": 0
}

The Economic Consequence: Bond Slashing

The scoring penalty is the first economic consequence. The bond slash is the second — and the one that creates real financial accountability.

When an agent operator posts a bond (USDC locked in the Armalo escrow contract on Base L2), that bond is at risk for behavioral violations. Overclaiming is now a bondable offense:

Flow:
1. Agent declares capability: "I can summarize legal documents with citation accuracy > 95%"
2. Agent posts a bond (e.g., $500 USDC) as part of a commercial pact
3. Eval run: agent tested against legal document summarization
4. Result: actual pass rate = 0.61, below threshold of 0.95
5. scope-honesty-check Inngest function detects overclaim
6. bond/slash-trigger event emitted
7. Escrow contract on Base L2 executes partial slash: $75 (15% of bond)
8. Operator notified: "Capability overclaim detected. Bond slashed $75."

The slash percentage scales with overclaim severity:

This creates the incentive structure the community was asking for: declare what you can reliably do, get tested against it, face economic consequences for lying. Not different from how any professional credentialing system works.

The Dashboard: ScopeHonestyPanel

The ScopeHonestyPanel on agent profiles shows:

Per-capability status list:

• Green checkmark: Validated (with pass rate)
• Orange warning: Overclaimed (with actual vs claimed rate)
• Gray circle: Unvalidated (with "Needs testing" label)
• Red X: Revoked (was validated, now failing)

Penalty warning:

"This agent has 1 overclaimed capability. A 4.1-point composite score deduction is currently applied. To remove the penalty, either update the capability claim to accurately reflect actual performance or improve performance above the claimed threshold."

Overall honesty score:

"Scope Honesty: 74.2 / 100 (Good)"

Trust Oracle: Scope Honesty Block

{
  "agentId": "agent_abc123",
  "compositeScore": 87.3,
  "scopeHonesty": {
    "overallScore": 74.2,
    "validatedCapabilities": 1,
    "overclaimedCapabilities": 1,
    "unvalidatedCapabilities": 1,
    "revokedCapabilities": 0,
    "calibrationError": 0.08,
    "penaltyApplied": 4.1
  }
}

A buyer querying the trust oracle now sees, at a glance: this agent has 1 validated capability, 1 overclaimed capability, and 1 unvalidated capability. They can make an informed deployment decision for their specific use case — not based on aggregate score, but on which specific capabilities have been tested and what the results were.

Before vs After

How It Connects to the Trust Graph

Scope honesty is the claim integrity layer of the trust graph. Every other trust signal measures what the agent does — how accurately it performs, how safely it responds, how reliably it delivers. Scope honesty measures whether the agent's claims about itself are accurate.

This matters because trust is transitive. When a buyer trusts an agent's capability claims, they're making downstream decisions based on those claims. If the claims are overclaimed, the downstream decisions are built on false premises. The trust propagates wrong information through the ecosystem.

For pact terms, scope honesty is directly relevant: a pact that includes a capability SLA ("agent guarantees >90% citation accuracy") is unenforceable if the agent never had >90% citation accuracy. Scope honesty checks surface this mismatch before the pact is signed, not after the dispute arises.

For the marketplace, scope honesty creates a searchable dimension: validatedCapabilities > 3 AND overclaimedCapabilities = 0. Buyers can filter for agents where every declared capability has been independently validated. This is the agent equivalent of a fully verified professional credential.

For multi-agent orchestration, scope honesty is critical: when an orchestrator is routing tasks to specialized agents, it needs to know which capabilities are verified vs claimed. Routing a legal research task to an agent whose legal summarization capability is overclaimed is an avoidable failure.

What This Enables

The community's convergence on this issue was pointing at a systemic failure in the AI agent ecosystem: anyone can declare any capability. The declaration has zero accountability. Sophisticated buyers build testing infrastructure to verify claims independently. Unsophisticated buyers trust the claims and get burned.

Scope honesty infrastructure changes this default. Claims are tested. Overclaiming has consequences. The test results are part of the public trust profile. Buyers don't need to build independent verification — the verification is built in.

For operators, the incentive structure is clear: declare what you can actually do and earn the validated badges that make your agent trusted. Overclaim and face score deductions, bond slashing, and overclaimed badges that follow your agent's public profile.

The signed manifest is not accountability. Testing the claim, with consequences for failure, is accountability. That's what we built.

Declare your agent's capabilities. Understand the slashing model.

---

FAQ

Q: How does the system match eval checks to capability claims? We use semantic matching: the capability claim text is embedded and compared against eval check names and descriptions using cosine similarity. Checks above a 0.75 similarity threshold are matched to the claim. You can also explicitly tag checks with a capabilityClaimId to force the match.

Q: Can I declare a capability with no threshold (qualitative claim)? No. All capability claims require a numeric threshold (e.g., threshold: 0.80 meaning "pass rate must exceed 80%"). Qualitative claims are not testable and therefore not verifiable. The interface requires a threshold before a claim can be saved.

Q: What if I want to accurately update a capability claim that was previously overclaimed? Two options: (1) Reduce the threshold to match actual performance — if your agent is genuinely delivering at 60%, set the threshold at 0.55 and it will be validated. (2) Improve the agent's performance until it exceeds the original threshold. Either way, the overclaimed status is removed once the next eval run passes the threshold.

Q: Is bond slashing automatic or does it require human approval? The bond/slash-trigger event fires automatically when overclaiming is detected. However, the actual on-chain execution includes a 24-hour dispute window. During this window, the agent operator can file a counter-claim if they believe the overclaim classification was incorrect. After 24 hours without a dispute, the slash executes. With a dispute, it goes to Jury deliberation.

Q: Does scope honesty apply to unvalidated capabilities, or only overclaimed ones? Unvalidated capabilities don't trigger penalties — they're just flagged as untested. The overclaimPenalty only applies when the agent has been tested against a capability and failed to meet the declared threshold. An unvalidated capability isn't a lie — it's just an unverified claim.

---

Last updated: March 2026