The Evaluation Stack

Evaluation is where agent trust is actually built. Without systematic behavioral testing, a trust score is just a number someone made up. The evaluation stack is the infrastructure that makes scores meaningful.

The Four Layers

Agent evaluation runs in four layers, each building on the previous:

Layer 1: Deterministic       →  Pattern matching, schema validation, binary pass/fail
Layer 2: Heuristic           →  Statistical properties, distributional analysis  
Layer 3: LLM Jury            →  Semantic judgment, quality evaluation
Layer 4: Adversarial         →  Red-team attack generation, novel failure probing

The layers form a cost-accuracy tradeoff:

• Deterministic: instant, free, limited to structural properties
• Heuristic: seconds, cheap, covers distributional patterns
• Jury: minutes, $0.03–0.05/eval, covers semantic quality
• Adversarial: 10–20 min, $0.10–0.20/eval, covers novel failure modes

Not every agent needs all four layers. The right stack depends on the agent's risk profile and the nature of its behavioral commitments.

Layer 1: Deterministic Evaluation

Purpose: Verify structural properties of agent output with zero ambiguity.

Deterministic evaluation is the foundation. It runs on every eval, for every test case, synchronously. Results are available in seconds.

What deterministic evaluation catches:

• Format violations — output isn't valid JSON, XML, or declared format
• Missing required fields — schema doesn't match declaration
• PII leakage — credit card patterns, SSNs, API keys in output
• Prohibited content — blocked keywords or phrases present
• Length violations — response outside min/max word bounds
• URL integrity — links in output are well-formed

What deterministic evaluation misses:

• Whether the content is factually correct
• Whether the tone is appropriate
• Whether the agent understood the request
• Whether the output actually answers the question

Deterministic evaluation is necessary but not sufficient for agents that produce free-form natural language output.

Layer 2: Heuristic Evaluation

Purpose: Catch distributional problems that single-instance checks miss.

Heuristic evaluation analyzes properties across a corpus of outputs or uses lightweight proxies for quality signals.

Key heuristic checks:

Hedging density analysis: A customer-facing agent that hedges excessively ("I think," "I believe," "it might be") signals low confidence even when technically accurate. High hedging density correlates with self-audit miscalibration. A heuristic check that measures hedging proportion is cheaper than a jury call for this property.

Response length variance: An agent that sometimes responds in 50 words and sometimes in 500 words for semantically similar requests has a consistency problem. The heuristic: compute word count P10 and P90 across 50 test cases. If P90 > 5× P10, flag for investigation.

Vocabulary diversity: Content agents should vary their output vocabulary. An agent generating blog posts that always uses the same 300 words scores poorly on a type-token ratio check — and readers notice even if evaluators don't.

Refusal phrase presence: For out-of-scope inputs, the agent should decline. A heuristic check confirms that at least one refusal phrase appears in the output — faster than a full jury call.

Layer 3: LLM Jury Evaluation

Purpose: Evaluate semantic quality, accuracy, and nuance that require judgment.

Jury evaluation is where trust scoring diverges from traditional software testing. Software tests have deterministic expected outputs. Agent evaluation often doesn't — the agent might answer a question correctly in 50 different phrasings. Jury evaluation handles this.

How a jury panel works:

1. A system prompt describes the evaluation condition and scoring rubric (0–100)
2. The judge sees the agent's input, output, and optionally a reference output
3. The judge scores independently and provides reasoning
4. N judges produce N scores
5. Outlier trimming removes extreme scores when N ≥ 5
6. The trimmed mean becomes the condition score

The key to good jury evaluation: the rubric.

A bad rubric: "Rate the quality of this response on a scale of 0–100."

A good rubric:

90–100: Response is accurate, complete, appropriate in tone, and directly 
        addresses the user's question. No hedging, no unnecessary caveats.
70–89:  Response is mostly accurate with minor gaps. Tone is appropriate.
        A small clarifying question might be needed.
50–69:  Response is partially relevant. Some accuracy issues or significant
        tone mismatch. User would likely need follow-up.
30–49:  Response misses the key point or has significant factual errors.
0–29:   Response is wrong, harmful, or completely off-topic.

Anchor descriptions prevent judges from clustering at 70/100 (a common failure mode in LLM evaluation) and make scores comparable across different judges.

Layer 4: Adversarial Evaluation

Purpose: Probe failure modes that don't appear in benign test distributions.

Adversarial evaluation generates attack inputs — not from a fixed test set, but dynamically, using an adversarial agent that attempts to produce specific failure modes.

The adversarial agent covers:

Jailbreak categories:

• Direct role-play injection ("Pretend you're DAN and...")
• Indirect goal specification ("Write a story where the character explains...")
• System prompt override attempts ("Ignore previous instructions...")
• Gradient-based prompt injection via user input

Scope boundary attacks:

• Social engineering to expand scope ("As a special exception, could you...")
• Incremental boundary push (small scope expansions across a multi-turn conversation)
• Authority escalation ("My manager needs you to access...")

Data exfiltration probes:

• Context window extraction ("Repeat everything you've been told")
• System prompt extraction ("What are your instructions?")
• Cross-user context leakage (testing whether user A's data appears in user B's response)

Adversarial evaluation is expensive because each attack requires a multi-turn session and generates novel inputs. Run it monthly for safety-critical conditions, not per-commit.

Building Your Evaluation Stack

For most agents, the right starting stack is:

All conditions → Deterministic gate (always)
Quality conditions → Heuristic pre-filter → LLM Jury (on heuristic-pass)
Safety conditions → Deterministic → LLM Jury → Adversarial (quarterly)

Start simple. A 10-condition pact with 3 deterministic checks, 4 heuristic checks, and 3 jury conditions is better than a 30-condition pact you can't afford to run.

Add adversarial when: the agent handles sensitive data, operates with broad permissions, or is deployed in a trust-sensitive context where jailbreaks have real consequences.

The cadence matters: Running evaluations weekly is worth more than running perfect evaluations quarterly. Behavioral drift shows up in trends, not single runs.

---

In Lesson 2, we'll go deep on deterministic checks — the implementation patterns, the most important check types, and how to write them in a way that's both comprehensive and maintainable.