Staging Evals Test a Snapshot. Production Is Every Single Call.

Evals exist. CI pipelines exist. Test suites exist. You verified before you shipped.

Then you shipped. And no one is watching.

Staging answers: did this agent pass verification before launch? It does not answer: is this agent passing verification right now, on this user's input?

These are different questions. Most agent pipelines answer only the first one.

A smoke detector in the factory does not tell you the house is on fire.

What post-deploy evals miss

Distribution shift. Your test set was built before you saw production traffic. The inputs users actually send differ from the ones you wrote. A pact violation in production does not look like your test cases — it looks like something you never anticipated.

Silent regressions. A model update, a prompt change, a dependency upgrade. None of it triggers your eval pipeline. Your score is stale. You find out when a user complains, not when the regression happened.

Latency contracts. Your eval measured latency once, under test load. Production latency is different — it varies by time of day, input length, upstream service health. If you have a latency SLA, point-in-time evals are not enforcing it.

No per-call record. When a violation happens, what was the input? What was the output? If you are not capturing every call, you are debugging from memory.

Wrap any existing agent call with production verification

import { ArmaloClient, createPactGuard } from '@armalo/core';

const client = new ArmaloClient({ apiKey: 'YOUR_API_KEY' });

// Attach a pact to any agent function — works with OpenAI, Anthropic, anything
const guard = createPactGuard(client, 'pact_abc123');

const result = await guard.call(userMessage, async (input) => {
  // your existing agent call — unchanged
  return await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: input }],
  });
});

// result.response = the OpenAI response, returned immediately
// result.verification = Promise → ValidationResult (runs in background)
result.verification.then((v) => {
  if (v?.passed === false) {
    console.warn('Pact violation:', v.violations);
  }
});

What you get: Every call captured — input, output, latency — verified against the pact contract in the background. Violations surface without adding latency to your response path. History accumulates into a composite score that reflects actual production behavior, not test-day behavior.

Staging tells you the agent was good at launch. Production tells you if it is still good now.

→ Get your API key: armalo.ai (free signup → API Keys) → Docs: armalo.ai/docs