Model Switching Makes Agent Evals Expire Faster Than Teams Think

If the action is low-risk, that may be acceptable. If the action changes money, customer state, security posture, or public communication, the fallback path matters.

The eval should not only say "the agent passed." It should say which model, tools, prompt, memory source, and routing policy passed. Otherwise, the proof is too abstract to govern runtime behavior.

Eval expiry triggers

The table is a practical review object. Every line is a reason to ask whether the old evidence still deserves to influence permission.

Route-aware trust scores

A route-aware trust score does not treat all successful runs equally. It asks which route produced the run and whether that route has current evidence for the task.

For example, a support agent may have excellent evidence on a premium model for refund triage but limited evidence on a cheaper fallback model for policy exceptions. If the runtime silently uses the fallback for a policy exception, the trust score should not inherit the premium route's confidence.

This sounds obvious once stated. In practice, many systems collapse the distinction because route metadata lives in observability while trust lives in governance. The two records need to meet.

Buyer diligence question

Buyers should ask vendors: does your trust evidence name the model route used in production?

If the answer is no, the buyer should treat the eval as a capability signal, not a runtime assurance. It may still be useful. It simply does not prove that the deployed agent path matches the tested path.

The Armalo route-evidence boundary

Armalo's trust architecture should make model route part of the evidence record behind pacts, scores, attestations, and recertification. That does not require claiming that one model is always safer than another. It requires treating route change as a proof-change event.

Today, Armalo's strongest public stance is that agents should earn trust through verifiable behavior and that trust should decay or narrow when evidence is stale. Model switching is a clean example of that doctrine. If the agent's model path changes, the permission supported by old evidence should be reviewed.

Practical operating pattern

Add route fingerprints to every high-risk agent run: provider, model, prompt version, tool schema version, retrieval bundle, and fallback reason. Then make eval validity depend on matching fingerprints or approved equivalence classes.

Equivalence classes are important. Teams should not re-run every eval for every harmless provider patch. But they should define which route changes are safe, which require targeted evals, and which force permission downgrade until proof returns.

The CFO version of the problem

Model switching is often introduced for cost reasons. That is reasonable. But cost savings should be measured after quality, incident, review, and recertification costs are included. A cheaper route that silently increases dispute rate or human review load may not be cheaper at all.

The finance team should therefore ask for route-level economics. What does each model path cost per successful accepted task? What is the rework rate? What is the escalation rate? Which route causes the most stale-source errors? Which route performs best under adversarial inputs? The answer may still favor a cheaper model for many tasks. The point is to make the tradeoff visible.

This gives engineering better language too. Instead of defending expensive models in abstract quality terms, the team can show where stronger evidence supports stronger authority. A premium route may be justified for settlement or security tasks while a frugal route handles drafting and classification.

The mistake is treating route choice as an implementation detail owned only by engineering. Route choice becomes a governance fact the moment it changes acceptance, escalation, or authority. A buyer does not need to inspect every token decision, but the buyer does deserve to know whether the trusted path and the production path are the same path.

FAQ

Does this mean model routing is unsafe?

No. Model routing is often necessary. The unsafe pattern is route opacity: treating all routes as if they inherited the same behavioral evidence.

What if the fallback model is better?

Then prove it. A better fallback should earn its own evidence and perhaps become the primary route. The issue is not quality; it is unearned authority.

Should trust scores be per model?

They should at least be route-aware. The meaningful unit may be model plus prompt, tools, memory, policy, and task class rather than model name alone.

The routing takeaway

Agent evals do not expire only on a calendar. They expire when the system being evaluated stops matching the system being run. Model switching makes that happen sooner than most teams admit.