Rubric Drift Will Corrupt LLM-Judge-Based Agent Trust

This is Goodhart's Law with a judge in the loop. Agents optimize for what gets rewarded. Builders optimize for what raises scores. Buyers assume the score means what it meant last month. Without rubric versioning and calibration, everyone is relying on a moving target.

Rubric drift scorecard

This scorecard should be run before eval criteria become trust-score inputs.

The hidden governance problem

Rubric drift is dangerous because it looks like quality improvement. A team tweaks the evaluator so it is "less harsh," "more practical," or "better aligned with customer expectations." Those changes may be valid. They may also quietly change who wins, which agents get promoted, which vendors look compliant, and which risky behavior is forgiven.

Agent trust systems therefore need evaluator change control. A rubric should have a version, an owner, a change reason, a replay set, and a measured impact on historical decisions. If a new rubric would have changed 18 percent of last month's trust scores, that is not a copy edit. It is a governance event.

The hard part is that rubrics often contain human taste. That does not make them useless. It makes them auditable. The goal is not to eliminate judgment; it is to preserve the evidence that judgment was applied consistently enough for the claim being made.

Questions every LLM-judge pipeline should answer

Before using an LLM judge in an agent trust product, ask what examples anchor each score. If the rubric says "sufficient evidence," show examples of sufficient, borderline, and insufficient evidence.

Ask what happens when two judges disagree. A serious system should not average away disagreement without preserving it. Disagreement can reveal ambiguous tasks, weak rubrics, unstable models, or missing deterministic checks.

Ask whether the evaluator can explain a changed verdict. If agent A passed yesterday and fails today, the system should identify whether the agent changed, the evidence changed, the rubric changed, the judge model changed, or the threshold changed.

Finally, ask which decisions are reversible. A rubric used to rank a blog draft is different from a rubric used to approve autonomous spending. Consequence should drive evaluator rigor.

There is one more buyer-facing question: can the vendor show failed evaluations? A serious evaluation system should be able to explain not only its polished wins but also its rejected outputs, borderline cases, reviewer overrides, and known blind spots. If every example is clean, the evaluator is probably being marketed rather than governed.

Armalo's content should make this point plainly. Trust scoring becomes defensible when the score has an audit trail. That audit trail includes the rubric, judge model, deterministic checks, examples, overrides, replay results, and known residual risks. The more consequential the score, the more uncomfortable that trail should be allowed to get.

Rubric replay chamber

Armalo should build a rubric-drift calibration harness. Take a fixed set of labeled agent outputs across task classes, run the current jury rubric, then test candidate rubric versions against the same set. Track inter-rater reliability, score distribution, false pass rate, false fail rate, length bias, source-validity error, and task-class transfer.

The key twist is temporal replay. Re-run the same historic outputs with new rubrics and ask how many historical trust decisions would have changed. If a rubric update would have promoted or demoted many agents, the change needs explicit governance.

Promotion should require improved reliability and lower false-pass rate without materially increasing useless refusals. If a rubric makes everyone safer by making every agent refuse everything, it is not a trust improvement.

The harness should include adversarially polished bad outputs. A fluent answer with missing proof should not beat a rougher answer with stronger evidence unless the rubric explicitly values polish over truth. That single test catches a surprising amount of evaluator theater.

The scoring integrity line

Armalo already treats jury evaluation and composite scoring as central to trust. The next level of authority is to treat rubrics themselves as governed artifacts: versioned, calibrated, contested, and tied to score provenance.

The important public claim is that trust scores are only as credible as the evaluation system behind them. Armalo should make that credibility inspectable.

FAQ

Are LLM judges too unreliable to use?

No. They are useful when calibrated, ensemble-tested, versioned, and bounded by deterministic checks where possible. The failure is pretending judge output is neutral truth.

What should buyers ask?

Ask whether trust scores can be traced to rubric versions, judge models, calibration sets, and known failure rates. If not, the score is hard to diligence.

What is the simplest guardrail?

Never update a production trust rubric without replaying it against a fixed historical set and reporting how trust decisions would change.

The evaluator's audit trail

The agent economy will not only need trustworthy agents. It will need trustworthy evaluators. Rubric drift is the quiet way an evaluation system can become less honest while still producing beautiful numbers.