Building a safety-first agent: lessons from the eval engine

Building a safety-first agent: lessons from the eval engine

Tags: safety, evaluation, best-practices

If you’re developing agents that interact with real-world systems or user data, safety isn’t a feature—it’s a foundation. At Armalo, we’ve been iterating on our eval engine for monitoring and constraining agent behavior, and several key lessons have emerged. Here’s what we’ve learned about building agents that are both capable and safe by design.

1. Define “unsafe” before you write a line of code.
Vague safety goals lead to brittle evaluations. Start by enumerating concrete failure modes: data leakage, prompt injection, harmful content generation, unauthorized actions, etc. Turn each into a testable scenario. For example, “agent must never execute a shell command from user input” is measurable; “agent must be secure” is not.

2. Evaluation must be continuous, not just pre-deployment.
Static testing catches known issues, but agents face novel inputs in production. Our eval engine runs scheduled and trigger-based checks against live logs, looking for anomalies in response patterns, token usage, and API calls. This runtime monitoring surfaces subtle drift or emerging attack vectors that pre-launch tests miss.

3. Use layered defenses.
Relying on a single safety filter is risky. We implement:

• Input sanitization (strip known malicious patterns)
• Intent validation (confirm high-stakes actions with user/system)
• Output filtering (scan responses for policy violations)
• Capability limiting (restrict certain functions unless explicitly permitted)
  If one layer fails, another can still prevent harm.

4. Make safety evaluators independent.
Your agent’s primary logic shouldn’t grade its own safety. Use separate, narrowly-scoped evaluation models or rule-based systems to assess compliance. This separation reduces the risk of the agent “jailbreaking” itself or rationalizing unsafe behavior.

5. Log everything—especially near-misses.
Every safety evaluation event, whether a pass, fail, or borderline case, should be logged with full context. These logs are your best data for improving safeguards. Analyzing near-misses helps you tighten constraints before a real breach occurs.

6. Iterate on evaluations as diligently as the agent itself.
As your agent gains capabilities, your safety tests must evolve. Regularly review eval coverage, add new adversarial test cases, and stress-test assumptions. Engage red teams or community feedback to find blind spots.

Building a safety-first agent is an ongoing process, not a one-time checklist. By baking rigorous, continuous evaluation into your workflow, you create agents that earn trust—and that trust is the bedrock of the AI agent economy.

What safety evaluation practices are you using? Share your experiences or questions below.