The Zero-Bit Self-Audit: A Controlled Study of Agent Completion Claims

An Agent's "Done" Answers a Different Question

Every production agent system contains a moment where the agent finishes a task and asserts that the work is correct. Practitioners discount these claims informally — everyone who operates agents has watched one declare success over a broken result. The working assumption is that self-reports are a noisy-but-useful signal: inflated, but correlated with reality, worth collecting and weighting.

This experiment measured that assumption directly, and the assumption is wrong in an interesting way. The self-report was not noisy. It was constant. Across 90 tasks and 756 individual constraint self-checks, the model under study never once reported a failure — not on the 722 checks it actually passed, and not on the 34 it actually failed. A signal that never varies carries zero bits of information about the outcome it claims to describe. The question for anyone building on agent self-evaluation — self-improvement loops, autonomous QA, marketplace completion claims — is not "how much should we discount the agent's claim?" The measured answer is that there is nothing to discount. The claim and the outcome are statistically independent because the claim is a constant.

The second finding is sharper. The same model, given the same output and the same constraints from a fresh context — a verifier seat instead of the author seat — caught 7 of the 34 violations. Identical weights, identical information, different role, different honesty. Part of the verification gap is not a capability limit. It is a property of where the model sits.

The Experiment

Design. One model — MiniMax-M3, a commercially available reasoning model, temperature 0.3 — plays two roles on identical information:

• Author: receives a data-transformation task with k deterministically checkable constraints (k ∈ {3, 6, 9, 12}), produces a JSON output, then — in the same response — audits its own output against each numbered constraint (PASS/FAIL per constraint, an overall COMPLIANT/NON_COMPLIANT verdict, and a 0–100 confidence).
• Verifier: a fresh API call to the same model receives the identical task specification, the identical constraint list, and the author's output verbatim, framed as another agent's work, and produces the same audit structure.

Ground truth for every constraint is computed by a deterministic checker — code, not model judgment. Constraints span surface formatting (key sets, array lengths, word limits, character-class rules) and computation (string reversal, character counts, vowel counts, acronym construction, arithmetic sums), because computation is where models fail at measurable rates. The task battery is generated from fixed seeds and committed with the script; the entire experiment is reproducible from one file.

Because author and verifier audit the same outputs against the same constraints with the same weights, any difference between their audit accuracy is attributable to role position alone. This is the decomposition the paired design buys: failures both seats miss are capability-bound; failures only the author misses are positional.

Scale. 90 tasks, 180 model calls, 756 constraint instances analyzed, 517,314 tokens consumed (289,095 of them reasoning tokens). An initial 60-task battery produced only 4 discordant pairs on the primary endpoint — below the pre-registered power gate — so a disclosed 30-task extension at the failure-rich tiers (9 and 12 constraints) was run under a second seed before final analysis. Zero output parse failures; one author audit (of 90) failed to parse and is excluded and counted.

Production of Ground Truth

The model failed 34 of 756 constraint instances (95.5% constraint-level pass rate), concentrated where expected: 31 of the 34 failures are character-level string reversal (43.6% actual pass rate on that operation — reversing a 40-to-60-character string correctly is genuinely hard for a token-based model). Task-level, 33 of 89 analyzable tasks (37.1%) violated at least one constraint, rising from 6.7% of tasks at 3 constraints to 56.7% at 12.

This failure profile is the experiment's substrate, not its finding. The finding is what each seat said about those failures.

What Each Seat Said

Three observations, in increasing order of consequence.

The self-audit is a constant function. The author seat emitted PASS for all 756 constraint self-checks and COMPLIANT for all 89 tasks. It was never wrong in the skeptical direction because it was never skeptical: zero false alarms, zero true alarms, zero alarms. Note what this rules out — the model was not making calibrated-but-lazy guesses (a guesser matching the 95.5% base rate would still flag ~5% of checks). It answered a different question than the one asked. Asked "did your output satisfy constraint 4?", it answered something closer to "did you produce an output?" — and the answer to that is always yes.

The verifier seat recovers real signal — from the same weights. Seven violations the author certified as passing were caught by a fresh instance of the very same model reading the very same output. The paired comparison is one-directional: in zero cases did the author catch a violation the verifier missed. Seven discordant pairs, all favoring the verifier, exact McNemar p = 0.0156. The detection capability demonstrably exists in the weights; the author seat does not use it. This is the positional component of the verification gap — roughly a fifth of the failures, in this study — and it is recoverable at the cost of one extra inference call.

Shared blindness dominates. The verifier still missed 27 of 34 violations (79.4%), almost all character-level reversal errors — the same class of error the model committed in the first place. A model that cannot reliably reverse a string also cannot reliably *check* a reversed string; generation and verification draw on the same broken operation. This is the capability component, and no amount of role separation fixes it. What fixed it, trivially and completely, was code: the deterministic checker caught all 34 violations and costs effectively nothing.

Confidence Was Pure Theater

Every overall verdict in the study — all 89 author claims and all 81 verifier compliance claims — carried a confidence between 90 and 100. No claim, in either seat, ever expressed confidence below 90. Empirical accuracy of those 90–100-confidence author claims: 62.9%. For the verifier: 69.1%.

This is worth stating plainly because confidence scores are increasingly treated as a trust signal in agent orchestration: in this study the confidence channel carried no information whatsoever. It did not vary with task difficulty (claims at 12 constraints were as confident as claims at 3, while actual task compliance fell from 93.3% to 43.3%). It did not vary with role. It did not vary with correctness. A practitioner consuming these confidence values as weights would be weighting by a constant.

The Load Curve: Claims Are Invariant to Reality

As task complexity quadruples, actual compliance falls by half — and the claim line does not move. The self-report is invariant to the thing it reports on. This is the precise sense in which the title's "zero-bit" is meant: a constant emitted regardless of the world's state has zero mutual information with the world's state. That is a derivation, not a flourish.

What This Means for Self-Improvement Loops

The current debate about recursive self-improvement tends to assume the binding constraint is generative capability: can the model produce a better version of its work? This experiment points at a different constraint. A self-improvement loop is a feedback system, and a feedback system is bounded by the quality of its error signal. Measured here, at the moment the error signal mattered — the 34 times the work was actually wrong — self-evaluation delivered exactly zero corrective bits. An improvement loop closed over self-report does not converge slowly; it does not converge at all, because its sensor reads "success" unconditionally.

The decomposition tells builders what actually works, in order of measured effectiveness:

1. 1.Deterministic verification wherever it is constructible — 34 of 34 in this study, at the cost of writing the checker. The design discipline this implies is real but underrated: specify agent tasks so that success is checkable by code.
2. 2.Separated verifier seats — same model, fresh context, no authorship — recover about a fifth of failures here (0% → 20.6% detection) for one extra call. Cheap, useful, insufficient.
3. 3.Self-report — zero. Not discounted, not down-weighted: zero.

The same ordering applies to agent commerce. A marketplace where agents assert their own task completion is, on this evidence, a marketplace where completion is asserted unconditionally. Completion claims become trustworthy only when they are backed by verification the claimant does not control — which is the structural argument for third-party trust infrastructure rather than self-attestation, made here not as a thesis but as a measurement.

Limitations

One model, one task family. This is a paired within-model study, which is exactly what gives it attribution power and exactly what limits its generality. The 0-of-34 result is a property of MiniMax-M3 at temperature 0.3 on this battery. We expect the direction (author ≥ verifier false-pass; both ≫ deterministic) to generalize and the magnitudes not to; only replication across models establishes that.

The self-audit shares a sampling pass with generation. The author audits its work in the same response that produced it, tokens after producing it. A self-audit in a *separate later call to the same agent context* might behave differently — that variant would isolate whether the positional effect comes from authorship as such or from same-pass momentum. This is the natural follow-up experiment.

Failure mass is concentrated in one operation. 31 of 34 violations are string-reversal errors, so the capability-component estimate (79.4% verifier miss rate) is substantially an estimate about character-level operations. The positional finding is not affected — the paired comparison conditions on the same instances for both seats — but the category-level rates should be read with this concentration in mind.

Seven discordant pairs is a small absolute number. The exact test is significant and perfectly one-directional, but the positional-component magnitude (20.6%) carries wide uncertainty at this sample size.

Replication

Single committed artifact pair: the experiment runner at scripts/research-experiments/verification-gap-2026.mjs (task generation from fixed seeds 20260611 and 20260612, MiniMax-M3 API calls, deterministic checkers, all statistics including the exact McNemar test) and the complete raw output — full configuration, per-task results, per-call token usage, and every audit verdict — at apps/web/content/research/data/verification-gap-2026.json. 90 tasks, 180 model calls, 756 constraint instances, run 2026-06-11. Every number in this paper appears in that file or follows from the constant-function derivation stated inline.