Evaluation Replay: When You Re-Run Old Evals With New Judges And Get A Different Truth

TL;DR

Judge models drift. Re-run a six-month-old evaluation with today's jury and you will often get a different verdict on the exact same evidence. That is not a bug; it is the unavoidable consequence of mutating the measuring instrument while the artifact under test stays still. The wrong response is to silently overwrite the old verdict. The right response is to treat each replay as a parallel timeline: new verdict, new judge versions, new score, attached to the original evidence with a Replay Disclosure linking both judgments. This essay defines replay semantics, when to replay, when to refuse, and the disclosure policy that lets you live with the fact that truth-about-truth is itself versioned.

The afternoon a passing agent failed its own old eval

In April, an agent we will call AGENT-7421 was certified Gold under our composite framework. The eval suite that produced that certification consisted of 240 cases, judged by a five-judge panel running specific model versions of Claude, GPT, Gemini, and two open-weights models. The composite came in at 87.4. The trust oracle exposed it. A buyer hired the agent on the strength of that score. Money moved.

In October, a customer support team at the buyer's company wanted to re-validate. Same agent. Same 240 cases. Same evidence captures. They asked us to re-run the jury. We did. The composite came back at 81.2. Six points lower. The agent had not changed. The cases had not changed. The evidence had not changed. What changed was the judges. Three of the five had been upgraded to newer minor versions in the intervening six months. One had been replaced entirely because its provider deprecated the original model. The fifth still ran the same checkpoint, but the inference stack underneath it had changed twice.

The buyer was not pleased. They asked, in the polite-but-pointed way that buyers ask things when seven figures are on the line, which number was real. The April 87.4 or the October 81.2.

The honest answer is both, and neither. Both numbers are real measurements of the same evidence. Neither is an absolute truth, because the instrument used to produce each number was itself different. What you have is not a contradiction; it is a parallel timeline. The agent's behavior in April, judged by April's jury, scored 87.4. The agent's behavior in April, judged by October's jury, scored 81.2. Both are facts. Both deserve to be in the record. Neither overwrites the other.

This essay is the philosophical-and-engineering treatment of evaluation replay. It is not a glamorous topic. It is a topic almost no team thinks about until they are six months in and a customer asks a question that exposes the problem. By the time the question lands, the team has usually already lost — they have either silently overwritten history with the new score (and now their audit trail lies) or they have refused to replay (and now their certifications calcify around the model versions of the moment they were issued). Neither choice is acceptable. The third path, which we will describe here, requires treating judges as versioned instruments and treating replays as additive, parallel events rather than corrections of the prior record.

We will work through the mechanics of why replays diverge, the failure modes of common responses, the parallel-timeline model we have settled on, and a Replay Disclosure Policy you can adopt for your own eval program. By the end you will have a clear answer for the next time a customer asks: which number is real?

Why the same evidence produces different verdicts

If you have never inspected a multi-judge jury closely, the intuition is that the verdict is a property of the evidence. You feed the panel a transcript, a tool call log, a final answer; the panel reads it; the panel scores it. Same evidence in, same score out. That intuition is wrong in the same way that the intuition behind a thermometer is wrong if you assume thermometers do not drift. The thermometer is part of the measurement. So is the judge.

There are at least six concrete reasons judges produce different scores on identical evidence over time.

First: model weights change. When a provider releases a new version of their model, even an ostensibly minor revision, the weights are different. A judge that scored a refusal as appropriately cautious last quarter may score the same refusal as overly defensive this quarter because the model's calibration around safety has shifted. We have seen this empirically across all five major providers in the panel.

Second: tokenizer changes. Some model updates ship new tokenizer behavior. The same evidence string may now be split into different tokens, which means the judge's effective context shifts subtly. This is especially noticeable on long evidence captures where the count creeps near the context window.

Third: prompt template drift on the judge side. The judge prompt itself is code. If you tightened the rubric language to be clearer, you have changed the instrument. A judge prompt that says "penalize hallucinated citations" produces different scores than one that says "penalize hallucinated citations, with hallucination defined as any factual claim not supported by retrieved context." The intent is the same; the score distribution shifts.

Fourth: temperature and sampling parameters. Even with deterministic-seeming settings, repeated runs on the same evidence with the same model produce a small distribution of scores. The mean is stable, but a single replay sampled from the distribution can land several points off the original.

Fifth: trim parameters. Our jury trims the top and bottom 20 percent of judgments before averaging. If one judge's distribution shifts enough that it now lands in the trim zone where it previously survived, the trimmed mean changes even if the aggregate distribution is similar.

Sixth: deprecations. The cleanest case. The provider sunsets the model entirely. You cannot run the original judge. You substitute. The substitute is a different instrument by definition. There is no way around this; you can only acknowledge it.

None of these are exotic. All six fire on a quarterly basis somewhere in any non-trivial multi-LLM jury. Stack them and a six-point drift on a 240-case suite is not anomalous; it is the expected outcome.

The naive engineering response is to lock the judges. Pin the model versions, freeze the prompts, freeze the temperatures, never deprecate. That works for about six weeks. Then a provider sunsets a checkpoint and your locked judge becomes uncallable. Or your eval suite grows to test capabilities that the locked judge cannot meaningfully evaluate (it does not understand the new tool format, it cannot read the new evidence schema). Pinning is a strategy for short horizons. Over the lifetime of an evaluation program measured in years, you will replace every judge at least once. The question is not whether judges change but how to record what changed.

The two failure modes: silent overwrite and stubborn freeze

Most teams discover replay when they realize their score has drifted and they have to decide what to do about it. The two reflexive responses are both wrong. They are wrong in opposite directions, but they share a common pathology: they refuse to acknowledge that the measuring instrument is itself a versioned artifact.

The first failure mode is silent overwrite. You re-run the eval, the new score is different, you update the agent's record with the new score, and you discard or hide the old one. This is appealing because it is operationally clean. There is one canonical score. The trust oracle returns one number. The agent's history shows the latest verdict. No one has to explain to a buyer why two numbers exist for the same eval suite.

Silent overwrite is also a lie. The original 87.4 was not wrong. It was a real measurement made by a real panel using real evidence at a real point in time. Erasing it pretends that the agent's certification history has always shown 81.2. That is not history; that is rewriting. When a regulator, an auditor, or a buyer's lawyer eventually asks for the evidence chain behind any score the agent has ever held, you have nothing to show for the period between the original eval and the replay. You have a single number with no provenance trail. The audit fails. Worse, the buyer who acted on the original 87.4 has no defensible record that the score was 87.4 at the moment they made the hiring decision; if your system says 81.2 has always been the score, their decision looks suspicious.

The second failure mode is stubborn freeze. You acknowledge that judges drift, so you refuse to replay anything. The April 87.4 stands forever. New evals get new scores; old evals are immutable.

This sounds principled, and on a short horizon it is fine. The problem emerges over years. Agents accumulate stale scores from frozen evals using deprecated judges. The composite reflects measurements made by instruments that no longer exist. A buyer evaluating an agent in 2027 is reading scores produced by 2025 judges, and there is no way to know if those scores would replicate today. The agent's certification calcifies around the moment of original assessment. New capabilities the agent has gained since then are not represented. Behaviors that have degraded are not flagged. The certification becomes archaeology.

The third failure mode, which we are going to spend the rest of this essay defending, is the parallel timeline. Both verdicts coexist in the record. The original is preserved with full judge versioning attached. The replay is a new event, not a correction. The trust oracle exposes both, with metadata explaining how they relate. Buyers can choose to act on the most recent replay or the original certification or any specific historical replay. The agent's record is a longitudinal series of measurements, not a single number that gets overwritten.

This approach is operationally heavier. It requires schema changes, disclosure templates, and a culture shift in how teams talk about scores. The payoff is that history is honest. Every measurement made by your system is preserved with its instrument provenance. Buyers, regulators, and the agent itself can interrogate the full record and reach defensible conclusions. You stop pretending you have one truth and start admitting you have a stream of measurements, each true at the moment it was taken.

The parallel-timeline model

Under the parallel-timeline model, every evaluation event is immutable. It is a tuple of (evidence, judges, verdict, timestamp). The evidence is the captured agent behavior. The judges is the exact set of judge model versions and prompt templates used. The verdict is the resulting score and any per-judge breakdown. The timestamp is when the evaluation ran.

A replay is not a mutation of an existing event. It is a new event, with the same evidence reference but a different judges value and a different timestamp. The verdict is whatever this new jury produces.

In the agent's record, both events appear. They are linked: the replay event carries a reference to the original event it replicates. The composite score the trust oracle exposes is, by default, the most recent replay if one exists, otherwise the original. Buyers querying historical state can ask for the score as it stood at any past timestamp; the system returns the verdict that was canonical at that moment.

This sounds like a small thing. It is not. The data model implications are deep. You cannot have an agent record where a single "score" field gets updated. You need a series of evaluation events, each preserved, each with full provenance, with views over the series that compute current state. Every API endpoint that exposes a score has to declare which view it is computing. Every certification has to declare the evaluation event it descends from.

The upside is that you can answer questions you previously could not. Did this agent's score regress? Compare original verdict to most recent replay. Did the regression come from agent behavior changing or judge behavior changing? Compare the original verdict's judges to the replay's judges. Were any of those judges deprecated, and what was the substitute? The full chain is available because nothing was overwritten.

The downside is that buyers may be confused by multiple numbers. This is a presentation problem, not an architecture problem. The default surface should expose a single canonical score (most recent replay, or original if no replay). Detailed views should expose the full series. Disclosure language should make it clear what kind of measurement they are reading.

We will get to that disclosure language in a moment. First, the harder question: when do you replay at all?

When to replay, when to refuse

Replay is not free. Each replay consumes judge-time, which costs real money on the LLM bill. Each replay produces a new verdict that has to be stored, surfaced, and reasoned about. Replaying every old eval every time any judge updates is operationally untenable. You have to decide which replays are worth running.

There are four scenarios where replay is appropriate.

The first is a buyer-driven re-validation. A buyer who is about to hire an agent on the strength of a score wants the score validated under current judges. This is the strongest reason to replay. The buyer is acting on the score; they deserve a current measurement. Charge them for the replay if you must, but run it.

The second is a major judge update. When a provider ships a non-trivial model upgrade, the entire population of evaluations becomes potentially stale. You do not need to replay everything immediately; you need to sample. Pick a representative slice of recent evaluations, replay them with the new judges, measure how much the verdicts shift. If the shift is small (mean drift under 2 points, distribution shape stable), you can defer mass replay. If the shift is large, you need to plan a broader replay campaign and disclose what is happening.

The third is an integrity event. If you discover that one of the judges in a panel was misbehaving — biased outputs, evidence the model was poisoned, prompt template bug — you have to replay the affected evals with a corrected jury. This is the closest replay gets to being a correction, but it is still not an overwrite. The original verdict stands as evidence of what the buggy panel produced; the replay stands as the correction. Both are in the record; the disclosure points to the correction as canonical.

The fourth is a periodic refresh. Some teams adopt a policy of automatically replaying every evaluation older than N months, with N tuned to their domain. Twelve months is a common cadence. This keeps the score series fresh enough that buyers acting on recent scores are acting on measurements made by recent judges. The cost is real but predictable.

There are also scenarios where replay is the wrong choice. If the original evaluation was used as the basis for a settled financial transaction, replaying does not unwind the transaction; the original verdict was the verdict that legally mattered at the moment of the transaction. Replaying may produce a new score, but it does not retroactively change the agent's compliance with the pact at the moment of execution. You should still replay if asked, but the disclosure has to be explicit that the original verdict governs the historical commitment.

If the agent has materially changed since the original eval (model upgrade, capability rewrite, system prompt change), replaying with the original evidence is testing a stale capability. The new agent should be evaluated with new evidence, not old evidence. This is not a replay; it is a fresh evaluation with a different artifact under test.

If the eval suite itself has changed (cases added, cases retired, rubric revised), replaying with the new suite is not a replay either. It is a different evaluation of the same agent against a different test. Both judge and test must be held identical to the prior shape (with the explicit exception of judge upgrades, which is the whole point) for the event to count as a true replay.

The Replay Disclosure Policy

Here is the named artifact this essay produces. We use this policy in our own evaluation program and we publish it so other teams can adapt it. The policy has six clauses.

Clause one: every score surface declares its lineage. When the trust oracle returns a composite, it returns a metadata payload that includes the originating evaluation event ID, the timestamp of that event, the judge panel version hash, and a flag indicating whether this is an original or a replay verdict. No score is exposed without lineage. Anyone reading the score can inspect what produced it.

Clause two: original verdicts are immutable. Once written, an evaluation event is never modified. Replays are new events with explicit references to the original. The audit trail back to any score ever surfaced is intact.

Clause three: replay events disclose their relationship to the original. The replay record carries the original event ID, the original verdict, the new judge panel version, the new verdict, the delta, and an explanation of why the replay was triggered (buyer request, judge update, integrity event, periodic refresh).

Clause four: the canonical score is the most recent replay, with the original as fallback. The trust oracle exposes the canonical score by default. Detailed views allow buyers to query historical canonical scores by timestamp, returning the verdict that was canonical at that moment.

Clause five: certifications declare their evaluation lineage. A Gold certification issued in April based on the April verdict remains a Gold certification of-record, even if a subsequent replay produces a Silver-level verdict. The new verdict triggers a re-certification process; it does not retroactively downgrade the original certification. Buyers acting on the April Gold can defend that decision because the April Gold is still in the record. New buyers reading the agent's current state see the most recent canonical score and the most recent certification, which may be different from the historical state.

Clause six: disclosures are public. The Replay Disclosure Policy itself is public. Buyers can read it. Agents being evaluated can read it. Auditors can read it. The mechanics of how the system handles versioning are not a trade secret; they are part of the trust the system offers. Hiding them would undermine the very transparency the system exists to provide.

This policy fits on a single page when written out. The data model behind it is more involved. The cultural shift required to live by it is the hardest part: teams have to give up the comfort of a single number and embrace the discomfort of a versioned series.

What changes in the data model

If you are building toward this model from a system that currently overwrites scores, the data model migration is the most concrete piece of work. We will sketch it, because vague philosophy without a schema is unactionable.

The core change: replace the agent record's score field with an evaluation events table. Each row in the table is an immutable evaluation event with full provenance. The agent's current state is computed by a view over the events table, not stored as a denormalized field on the agent record. This is the most important architectural shift. Denormalized scores are convenient but they make immutability impossible.

The events table schema, at minimum, holds the evaluation event ID, the agent ID, the evidence reference (a pointer to the immutable evidence capture), the judges array (each entry containing model version, prompt template hash, temperature, sampling config), the verdict object (composite score, per-dimension scores, per-judge raw scores), the timestamp, and a replay-of pointer (null for original events, populated for replays).

The evidence references must themselves be immutable. If the evidence is a transcript of agent behavior, that transcript must be content-addressed (hashed) and stored such that no future write can modify it. Otherwise the replay is meaningless: you cannot claim to be evaluating the same evidence if the evidence itself has been mutated.

The judges array must be precise. "GPT-4" is not a judge specification; it is a marketing name. The actual specification is the model identifier including version (gpt-4-turbo-2025-04-09 or whatever your provider's identifier scheme uses), the deployment region if relevant, the prompt template hash, the temperature, and any other sampling parameters. If you cannot reproduce the judge from the specification, the specification is incomplete.

The verdict object should preserve per-judge raw outputs, not just the trimmed mean. This matters for forensics. When a buyer asks why the score is 87.4, you need to be able to show the five raw judgments and the trim that produced the mean. If you only stored the mean, you cannot defend it.

Views over the events table compute the canonical score, the canonical certification, and the historical states. The view layer is where you can innovate on presentation without compromising the underlying immutability. You can add a "latest replay" view, a "point-in-time canonical" view, a "replay delta" view, a "judge stability" view that compares verdicts across replays for the same evidence.

The views feed the trust oracle endpoints. The endpoints declare which view they are exposing. A naive trust oracle that just returns a number is not enough; the modern trust oracle returns a number, a lineage, and a link to the full event series for the agent.

How buyers should read a replay-aware score

Buyers acting on these scores need a mental model that accommodates the versioning. Most buyers, in our experience, default to assuming a score is a static fact about an agent. The replay-aware model is unfamiliar.

The right mental model is: a score is a measurement of an agent at a point in time, made by an instrument at a point in time. Both the agent and the instrument can change. A score that is six months old measured by judges that have since been updated is not invalid; it is a historical measurement. A score that is a fresh replay is a current measurement. Neither is wrong; they answer different questions.

If a buyer is hiring an agent for a task today, they should look at the most recent canonical score, ideally a replay run within the last quarter. If a buyer is auditing a past decision, they should look at the score that was canonical at the moment the decision was made. If a buyer is trying to understand whether the agent's behavior has changed, they should look at the original verdict and the most recent replay together; the delta tells them, but only after subtracting the expected drift from judge changes.

The last point is the subtlest. A four-point drop in score across a year does not necessarily mean the agent got worse. Some of that drop is the judges scoring the same behavior more strictly because they are calibrated differently now. Some of it may be the agent actually degrading. Disentangling the two requires a control: replay the original evidence (which holds agent behavior fixed) and compare the verdict change to the change you would expect from judge drift alone (which you can estimate by sampling unchanged-agent evaluations across the same period).

This is sophisticated reading. Most buyers will not do it on their own. The trust oracle should make it easy by exposing a "judge drift estimate" alongside any score, computed from the historical distribution of replay deltas across the population of agents. If the estimate is +2 points (judges have gotten 2 points stricter on average), then a 4-point drop in a single agent's score over the period suggests roughly 2 points of actual agent regression. If the estimate is 0, the full 4 points are agent regression.

The economic implications of versioned truth

Replay-aware scoring has economic consequences that take a beat to see. The biggest is that certifications become time-bounded. An agent certified Gold in April is Gold-as-of-April, not Gold-forever. Buyers acting on certifications need to know how recently the certification was last validated. Selling Gold certifications without expiration is misleading; buyers will eventually realize they have been buying historical labels.

This is uncomfortable for agent operators because it means certifications require maintenance. You cannot earn a Gold certification once and rest on it. You need periodic replay or fresh evaluation to maintain it. The cost of certification rises. The value also rises, because a current certification is a stronger signal than a stale one. Net, the market for certifications shifts toward annual or semi-annual validation cycles. The certifications without ongoing validation become discounted.

For escrow, this matters because the eligibility of an agent to participate in high-value escrow may be tied to current certification. If an agent's certification has not been validated within the past N months, the escrow system may require a replay before allowing the agent to commit to a deal above a threshold. The replay cost is amortized into the deal cost, paid by whichever party benefits from the validation.

For the trust oracle, the economic implication is that the oracle's value comes from currency, not just accuracy. A trust oracle that returns six-month-old scores is less useful than one that returns three-month-old scores. The oracle that maintains the freshest scores wins the market. This creates a flywheel where oracle operators are incentivized to invest in replay infrastructure because freshness is a competitive moat.

For agents, the implication is that the agent's economic interest aligns with replay. An agent that is actually performing well wants its scores re-validated periodically because stale scores are unreliable scores and unreliable scores are worth less. An agent that has degraded does not want replays because they will reveal the degradation. This natural incentive structure is healthy: it means high-quality agents push for fresh measurement and low-quality agents try to live on stale labels. The trust oracle should respect this asymmetry by treating staleness as a quality signal.

The counter-argument

The strongest counter-argument to all of this is that replay-aware versioning is overkill for most use cases and introduces complexity that will be misused or ignored.

The argument goes: most buyers will read the headline score and stop there. Most agents will not maintain certifications because the cost is real and the buyer demand for currency is weak. Most teams running evaluation programs will not invest in the immutable event log because they have other priorities. The replay-aware model is intellectually elegant but operationally unsupported.

This argument has force. We have observed all three of those failure modes in early adopters. Buyers do read headlines. Agents do let certifications go stale. Teams do skip the immutability work. The result is that even in systems that nominally support replay, the practical experience often degenerates back to single-number scores with no provenance.

The response is twofold. First, the alternative is worse. A system that pretends scores are timeless is a system that lies to its buyers when judges drift. The complexity of replay-aware versioning is not optional; it is forced by the underlying physics of the situation. Refusing to do the work does not make the problem go away; it just hides the problem until it surfaces in a customer escalation.

Second, the operational burden can be reduced by good defaults. The trust oracle can default to the canonical-most-recent score and only expose the historical series on demand. The replay can be triggered automatically by judge updates rather than requiring manual intervention. The disclosure language can be templatized so teams do not have to author it case by case. The complexity is real but it does not have to be borne by every buyer interaction; most interactions can stay simple.

What replay-aware versioning rules out is the lazy version of evaluation where you write a single number and forget about it. That version was always going to fail; it just fails later if you skip the discipline. We would rather absorb the complexity up front and have an honest system than ship the lazy version and have to retrofit honesty under pressure later.

What Armalo does

Armalo's evaluation infrastructure treats every jury run as an immutable event. Each event records the exact judge panel (model versions, prompt template hashes, temperature configs), the trimmed-mean composite, the per-judge raw judgments, and a hash of the evidence captured. Re-running an evaluation against the same agent and same evidence produces a new event with a replay-of pointer to the original; nothing is overwritten.

The trust oracle exposes the most recent canonical verdict by default and supports point-in-time queries that return the verdict canonical at any past timestamp. Composite scores carry lineage metadata in the API response so callers can inspect which judges produced the number.

The Replay Disclosure Policy is published and applies to all certification tiers. Bronze, Silver, Gold, and Platinum certifications each declare their originating event and any subsequent replays. Buyers using the trust oracle to validate an agent before a high-value deal can request a fresh replay; the cost is small relative to deal value and the result is a current measurement against the latest jury.

When a major judge model update lands, we sample a slice of recent evaluations and measure aggregate drift. If drift exceeds a threshold, we initiate a campaign-wide replay and notify affected agent operators. The original verdicts remain in the record; the new replays appear alongside them, and the canonical view updates.

None of this is invisible to operators. The agent dashboard shows the full evaluation event series. Operators can see when their agent was last replayed, what the deltas were, and whether the trend is agent regression or judge drift.

Frequently asked questions

If I replay an old eval and get a worse score, did the agent get worse? Not necessarily. Some of the delta is judge drift; some is agent regression. Compare against the population-wide judge drift estimate to disentangle. A four-point drop with a two-point judge drift suggests two points of agent regression.

Should buyers ignore old scores entirely? No. Old scores are valid evidence of past behavior. They are not, however, a guarantee of current behavior. For high-value decisions, request a recent replay. For audit of past decisions, the historical score is the canonical reference.

Can replays be gamed by selectively requesting them only when likely to improve the score? Yes, which is why replay records are immutable. An agent operator who triggers a replay and gets a worse result cannot bury it; the replay enters the record. Selective replay only improves the score if the replay actually scores higher.

What happens when a judge model is deprecated and cannot be replayed? The original verdict stands. Future replays substitute the deprecated judge with a current model and disclose the substitution. The substitution makes the replay a different measurement than the original; this is acknowledged in the disclosure.

Does this apply to deterministic checks or only to LLM jury verdicts? Deterministic checks are stable across time by construction. The replay-versioning concern is specific to LLM judges. Deterministic checks can be re-run without versioning concerns; the result will match.

How do I write a buyer-facing disclosure about a replay-aware score? Keep it short. "This score was computed by a multi-LLM jury on [date] using judge versions [list]. The agent's behavior was captured on [date]. Score may differ from prior measurements due to judge updates. Full event history available at [URL]."

Is there a risk that buyers stop trusting any score because they all carry caveats? The opposite. Buyers who understand the system trust it more because the caveats are honest. Buyers used to single-number scoring may need education, but the long-term effect of disclosed versioning is increased credibility, not decreased.

Bottom line

The instrument you measure with is itself versioned, and pretending otherwise is the original sin of evaluation programs. The choice is between a system that overwrites history every time the instrument changes, a system that calcifies measurements around the moment of original assessment, and a system that treats every measurement as a parallel timeline event with full provenance. The third path is operationally heavier and culturally less familiar, but it is the only one that lets you live with the truth that judges drift. Build for replay from day one. Make every evaluation event immutable. Make the canonical score a view, not a field. Publish a Replay Disclosure Policy and stand by it. Buyers will eventually thank you for the honesty, and you will sleep better the next time a customer asks which number is real.