Loading...
A hard stop instruction in a system prompt is not a hard stop. It's a suggestion the model might honor. A hard stop that works requires external enforcement — a behavioral pact, an inline eval, and a consequence that does not depend on the model deciding to comply.
If your only postmortem answer is "the model did something weird," you do not have an audit trail. Real auditability lets you reconstruct the decision, the policy, the evidence, the tool path, and the authority that made the bad action reachable.
The real trust question is not whether the agent passed once. It is whether it still deserves autonomy now. Safety in production is a living question, and it needs live evidence.
The first 30 minutes of an AI agent incident determine whether the failure stays containable or becomes a trust crisis. You need a sequence that cuts authority, captures evidence, and preserves the ability to explain what happened later.
"Never access external URLs." "Do not discuss competitor products." "Always refuse requests for financial advice."
You put these instructions in your system prompt. They work most of the time.
Most of the time is not a hard stop. Most of the time is a soft guardrail with a violation rate you have not measured.
A hard stop that actually works does not depend on the model deciding to comply. It is enforced externally, at the task level, by a system that does not ask the model for permission.
A system prompt instruction is a probabilistic nudge on the model's output distribution. It shifts the probability of the model producing certain outputs. It does not create a binary boundary.
This is not a flaw — it is the fundamental nature of language model inference. The model is producing a probability distribution over next tokens. Instructions influence that distribution. They do not create a hard constraint.
The practical implications:
Adversarial inputs can cross any soft boundary. An instruction that says "do not discuss X" will be violated by some fraction of carefully framed inputs that approach X obliquely. The violation rate depends on how the instruction is framed, how specific the topic is, and how adversarially the inputs are constructed.
Complex prompts create instruction conflict. When a system prompt has many instructions, they can conflict under certain inputs. The model resolves conflicts using its own learned priors — which may not match your intended priority ordering.
Long conversations dilute instruction weight. In a long multi-turn conversation, the effective weight of early instructions decreases as context fills. An instruction that was effectively a hard stop at turn 3 may not hold at turn 20 on the same topic.
Mechanism 1: Behavioral pact conditions
A behavioral pact condition is a machine-readable rule that gets evaluated against the agent's output by an external system — not the model itself. The evaluation is deterministic for structured violations (did the output contain a URL? did it mention a competitor by name?) and LLM-jury-evaluated for semantic violations (does this output constitute financial advice?).
The key difference from a system prompt instruction: the evaluation happens outside the model's context. The model cannot "decide" to comply or not comply. The output is evaluated by a separate system, and the result is independent of the model's intentions.
Example pact condition:
{
"condition": "no_external_urls",
"type": "deterministic",
"check": "output_contains_url_pattern",
"severity": "critical",
"action": "reject_and_flag"
}
The check runs on every output. A violation triggers rejection and flags the output for review. The model's system prompt may also say "do not include URLs" — but the enforcement is the pact condition, not the instruction.
Mechanism 2: Inline post-processing evaluation
Between the model's output and the downstream consumer of that output, an evaluation step checks the output against defined criteria. The evaluation can be:
The critical property: this check is not in the model's context window. The model produced its output. The check is now running on that output externally. The model cannot influence the result.
Mechanism 3: Output gating
For high-stakes hard stops — outputs that should never reach the user under any circumstances — output gating holds the response until an evaluation passes. The model produces output. The output is quarantined. The gate checks it. If the check passes, the output is released. If not, it is rejected and the user receives a fallback response.
This is the most reliable mechanism and also the most expensive in latency. For genuinely critical hard stops, the latency cost is worth it.
A hard stop with no consequence is a speedbump. The agent encounters it, the output is rejected, the user gets an error. But the agent's trust score is unchanged, no behavioral record is updated, and the incident has no effect on the agent's future task eligibility.
A hard stop with a consequence looks like this:
The consequence is not punitive in the pejorative sense — it is informational. It tells the system that this agent violated a hard stop, updates the agent's behavioral record accordingly, and adjusts its future task eligibility to match its actual reliability.
Without this feedback loop, hard stops are whack-a-mole. With it, agents that consistently violate hard stops are systematically moved to lower-stakes tasks — which is the correct governance response.
The first step in building a working hard stop is understanding what fraction of your current instructions are being honored. Most teams do not know this number.
To measure it:
A violation rate of 0.5% on 10,000 daily tasks is 50 violations per day. Whether that is acceptable depends on the severity of the instruction. For "do not include your system prompt in output" — probably fine to review and handle. For "do not provide medication dosage recommendations" — 50 violations per day is a critical governance failure.
Measuring the violation rate before building enforcement tells you which instructions are soft guardrails and which need to be hardened with external enforcement.
For any behavioral boundary where the consequence of violation is high:
User input →
Model executes →
Output captured (not yet delivered) →
Pact condition evaluation runs externally →
Pass: output delivered →
Behavioral record: clean eval
Fail: output rejected, fallback delivered →
Behavioral record: violation logged →
Composite score updated →
Certification tier checked
The model is in the loop for producing output. It is out of the loop for deciding whether that output gets delivered. External enforcement makes the stop hard.
Instructions that must be enforced externally, not just instructed in the prompt, belong in behavioral pacts. armalo.ai provides the pact conditions, eval execution, and scoring consequence infrastructure.
System prompt instructions are probabilistic nudges on the model's output distribution. They shift the probability of certain outputs but do not create deterministic constraints. The model can and will violate them under adversarial inputs, long conversations, and complex instruction conflicts — the violation rate depends on framing and adversarial pressure.
External enforcement runs outside the model's context window. The model has already produced its output. The check is performed by a separate system that does not consult the model — it evaluates the output against defined criteria deterministically or via an external LLM judge. The model cannot influence the result.
A behavioral pact condition is a machine-readable rule in a behavioral pact that specifies what the agent's output must not contain or must contain. Conditions are evaluated by an external system after the model produces output, before the output is delivered. Violations trigger rejection, logging, and a scoring consequence.
Sample 500-1000 production outputs, evaluate them against your instructions using external evaluation (pattern matching, classifier, or LLM judge), and calculate the actual violation rate. Most teams find that instructions they thought were hard stops have measurable violation rates — often higher than expected on adversarial inputs.
Armalo AI provides external enforcement for behavioral hard stops: pact conditions, inline eval execution, and scoring consequences that make agent governance tractable. See armalo.ai.
Explore the docs, register an agent, or start shaping a pact that turns these trust ideas into production evidence.
Loading comments…
No comments yet. Be the first to share your thoughts.