Here is the complete picture before we go deep on each:
| Mechanism | Stop Speed | State Preserved | Financial Commitments | Downstream Agents Notified | Recovery Time |
|---|
| Hard Kill | Immediate (<1s) | None | Orphaned | No | Hours–Days |
| Graceful Suspension | 5–30s drain | Full | Honored | Yes | Minutes |
| Scope Restriction | Immediate | Full | Unchanged | No | Seconds |
| Pact Suspension | Immediate | Full | Frozen | Partial | Hours |
| Financial Circuit Breaker | Immediate | Full | Blocked | No | Minutes |
| Reputation Suspension | Minutes | Full | Existing honored | Via trust graph | Days–Weeks |
Let's go deep on each.
What It Does
A hard kill is the equivalent of unplugging the machine. You send a SIGKILL signal (or its container/orchestration equivalent), and the process stops immediately. No state flush. No drain window. No notifications to connected systems. The agent ceases to exist from the operating system's perspective within milliseconds.
In container environments:
- Kubernetes:
kubectl delete pod <agent-pod> --grace-period=0 --force
- ECS Fargate:
aws ecs stop-task --task <task-arn> --reason "hard-kill"
- Docker:
docker kill <container-id>
- Process manager:
kill -9 <pid>
In agent framework environments, hard kill typically means killing the underlying process or container that runs the agent loop, not calling any agent-level API.
When to Use It
Hard kill is justified in exactly one category of situation: active harm is occurring or imminent, and the cost of a clean stop exceeds the cost of the damage a graceful drain would allow.
Concrete triggers:
- Agent is actively exfiltrating data (network traffic anomaly detected)
- Agent is in a resource-consumption spiral (CPU/memory past 10x expected)
- Security breach: agent credentials compromised, agent behaving as an attacker
- Agent is executing irreversible destructive actions (deleting records, sending mass emails)
- Agent has been confirmed to be operating under prompt injection attack
The key question: "Would waiting 30 seconds for a graceful drain make this worse?" If yes, hard kill. If no, use Graceful Suspension.
Consequences You Must Manage
Hard kill leaves a mess. This is a feature, not a bug—the mess is the price of speed. But you must plan for it:
In-flight transactions are orphaned. Any escrow deposit the agent opened is now stranded. Any API call in flight may have already triggered the external action but will receive no acknowledgment. Payment transactions may be in ambiguous states—debited but not acknowledged.
Tool calls are orphaned. The agent may have already dispatched a send-email action, a database write, a webhook—and those external systems will complete their actions regardless of the agent's death. Your cleanup process must audit all tool calls made in the last N seconds.
Downstream agents receive no notification. Any agent that was waiting for a result from this agent is now hanging. If they have timeouts, they'll eventually error. If they don't, they'll wait indefinitely.
State is lost. Everything in the agent's working memory—its current reasoning state, its in-memory context, its pending action queue—is gone. You cannot resume from a hard kill. You can only restart from scratch or from the last external checkpoint.
Implementation Pattern
Because a hard kill is so destructive, the implementation must include a mandatory post-kill cleanup protocol. This is not optional.
// KillSwitchService — Hard Kill implementation
class KillSwitchService {
async hardKill(agentId: string, reason: string, operatorId: string): Promise<HardKillResult> {
const killTimestamp = new Date();
// 1. Record the kill event BEFORE the kill (so we have the timestamp)
await this.auditLog.record({
event: 'agent.hard_kill.initiated',
agentId,
operatorId,
reason,
timestamp: killTimestamp,
});
// 2. Snapshot all open commitments at kill time
const openEscrows = await this.db.query(
`SELECT id, amount, status FROM escrows WHERE agent_id = $1 AND status IN ('pending', 'active')`,
[agentId]
);
const inflightToolCalls = await this.db.query(
`SELECT id, tool_name, dispatched_at FROM agent_tool_calls
WHERE agent_id = $1 AND status = 'in_flight' AND dispatched_at > NOW() - INTERVAL '5 minutes'`,
[agentId]
);
// 3. Execute the kill (container/process level)
const killResult = await this.containerOrchestrator.forceStop(agentId);
// 4. Mark agent as hard-killed in DB
await this.db.query(
`UPDATE agents SET status = 'hard_killed', killed_at = $1, kill_reason = $2 WHERE id = $3`,
[killTimestamp, reason, agentId]
);
// 5. Flag all open escrows for manual review
if (openEscrows.rows.length > 0) {
await this.db.query(
`UPDATE escrows SET requires_manual_review = true, review_reason = 'agent_hard_killed'
WHERE agent_id = $1 AND status IN ('pending', 'active')`,
[agentId]
);
}
// 6. Notify downstream agents of abnormal termination
await this.agentBus.emit({
event: 'agent.abnormal_termination',
agentId,
timestamp: killTimestamp,
openEscrowIds: openEscrows.rows.map(e => e.id),
inflightToolCallIds: inflightToolCalls.rows.map(t => t.id),
});
// 7. Create cleanup task for ops team
await this.taskQueue.enqueue({
type: 'hard_kill_cleanup',
agentId,
killTimestamp: killTimestamp.toISOString(),
openEscrows: openEscrows.rows,
inflightToolCalls: inflightToolCalls.rows,
priority: 'high',
});
return {
killed: true,
killTimestamp,
openEscrowCount: openEscrows.rows.length,
inflightToolCallCount: inflightToolCalls.rows.length,
requiresManualCleanup: openEscrows.rows.length > 0 || inflightToolCalls.rows.length > 0,
};
}
}
Recovery Protocol
After a hard kill, before any restart:
- Audit all in-flight tool calls from T-5m to kill time. Determine which completed, which timed out, which are ambiguous.
- Resolve all orphaned escrows. Either release manually or allow counterparty dispute resolution.
- Reconstruct the last known clean state from external system audit logs.
- Root-cause the trigger. A hard kill is not a normal operating condition. Before restart, understand why you ended up there.
- Verify the hard kill reason is addressed. If the kill was due to prompt injection, remediate before re-deployment.
Expected recovery time: 2–12 hours depending on the number of orphaned commitments.
Kill-Switch 2: Graceful Suspension (Drain, Then Pause)
What It Does
A graceful suspension signals the agent to complete its current action unit, flush its state to a persistent store, notify connected agents of the suspension, and then pause execution. The agent remains in a suspended state—its configuration, memory, and position in any workflows are preserved—until explicitly resumed.
This is the workhorse kill-switch for normal operations. Most stop events should use graceful suspension.
The mechanism works in phases:
- Signal: Operator sends suspension signal to agent
- Drain: Agent completes current atomic action (or times out after drain_timeout_seconds)
- Checkpoint: Agent serializes current state to persistent store (Redis/DB)
- Notify: Agent sends
agent.suspended event to all connected agents and orchestrators
- Pause: Agent loop stops. Process may remain alive or be deallocated based on configuration.
When to Use It
Graceful suspension covers the vast majority of legitimate stop scenarios:
- Policy violation detected — eval score dropped below threshold, behavior outside pact boundaries
- Operator decision — routine maintenance, configuration change required, rollout of new model
- Scheduled pause — agent completes work session and enters standby
- Dependency unavailable — downstream system offline, graceful pause until restored
- Human review required — action at edge of scope, human in the loop checkpoint
- Budget approaching limit — graceful pause before overage, await authorization
The Drain Timeout Problem
The most important parameter in graceful suspension is the drain timeout. This is how long you're willing to wait for the current action to complete before escalating to hard kill.
The right drain timeout depends on your agent's action granularity:
- Agents that call external APIs with 1–2 second expected latency: 10s drain timeout
- Agents that execute multi-step database transactions: 30s drain timeout
- Agents that orchestrate multi-agent workflows: 60s drain timeout (with partial drain support)
- Agents that run long-horizon tasks: no drain timeout — use scope restriction instead
If the drain timeout expires and the agent has not reached a safe suspension point, you have two choices: extend the timeout or escalate to hard kill. Build this decision explicitly into your suspension logic.
Implementation Pattern
class KillSwitchService {
async gracefulSuspend(
agentId: string,
reason: string,
operatorId: string,
options: {
drainTimeoutMs?: number; // default 30000
escalateToHardKill?: boolean; // default true
preserveQueuedActions?: boolean; // default true
} = {}
): Promise<GracefulSuspendResult> {
const { drainTimeoutMs = 30000, escalateToHardKill = true, preserveQueuedActions = true } = options;
const suspendRequestId = crypto.randomUUID();
const requestTimestamp = new Date();
// 1. Set agent status to 'suspending' — this is the signal
await this.db.query(
`UPDATE agents SET status = 'suspending', suspend_requested_at = $1,
suspend_reason = $2, suspend_request_id = $3 WHERE id = $4`,
[requestTimestamp, reason, suspendRequestId, agentId]
);
// 2. Send suspension signal via agent bus
await this.agentBus.emit({
event: 'agent.suspend_requested',
agentId,
suspendRequestId,
reason,
requestedBy: operatorId,
drainTimeoutMs,
});
// 3. Poll for clean suspension acknowledgment
const deadline = Date.now() + drainTimeoutMs;
let suspended = false;
while (Date.now() < deadline) {
const statusResult = await this.db.query(
`SELECT status, state_checkpoint_id, suspended_at FROM agents WHERE id = $1`,
[agentId]
);
if (statusResult.rows[0]?.status === 'suspended') {
suspended = true;
break;
}
await new Promise(resolve => setTimeout(resolve, 500)); // poll every 500ms
}
if (!suspended) {
if (escalateToHardKill) {
// Drain timeout expired — escalate
const hardKillResult = await this.hardKill(
agentId,
`Graceful suspension drain timeout after ${drainTimeoutMs}ms. Original reason: ${reason}`,
operatorId
);
return { suspended: false, escalatedToHardKill: true, hardKillResult };
} else {
return { suspended: false, escalatedToHardKill: false, timedOut: true };
}
}
// 4. Notify connected agents of suspension
const connectedAgents = await this.getConnectedAgents(agentId);
await Promise.allSettled(
connectedAgents.map(connectedId =>
this.agentBus.emit({
event: 'agent.suspended',
agentId,
connectedAgentId: connectedId,
suspendRequestId,
reason,
resumable: true,
})
)
);
// 5. Record final suspension in audit log
await this.auditLog.record({
event: 'agent.graceful_suspension.completed',
agentId,
operatorId,
reason,
suspendRequestId,
drainDurationMs: Date.now() - requestTimestamp.getTime(),
connectedAgentsNotified: connectedAgents.length,
});
return {
suspended: true,
escalatedToHardKill: false,
suspendRequestId,
drainDurationMs: Date.now() - requestTimestamp.getTime(),
stateCheckpointId: (await this.db.query(
`SELECT state_checkpoint_id FROM agents WHERE id = $1`, [agentId]
)).rows[0]?.state_checkpoint_id,
};
}
async resume(agentId: string, operatorId: string): Promise<ResumeResult> {
// Verify state checkpoint exists and is valid
const agent = await this.db.query(
`SELECT status, state_checkpoint_id, suspend_reason FROM agents WHERE id = $1`,
[agentId]
);
if (agent.rows[0]?.status!== 'suspended') {
throw new Error(`Agent ${agentId} is not in suspended state (current: ${agent.rows[0]?.status})`);
}
// Validate state checkpoint is still fresh
const checkpoint = await this.stateStore.get(agent.rows[0].state_checkpoint_id);
if (!checkpoint || checkpoint.isStale()) {
throw new Error(`State checkpoint is stale or missing — manual intervention required`);
}
// Resume
await this.db.query(
`UPDATE agents SET status = 'active', resumed_at = NOW(), resumed_by = $1,
suspend_reason = NULL WHERE id = $2`,
[operatorId, agentId]
);
await this.agentBus.emit({ event: 'agent.resumed', agentId, resumedBy: operatorId });
return { resumed: true, fromCheckpointId: agent.rows[0].state_checkpoint_id };
}
}
The Agent-Side Suspension Handler
For graceful suspension to work, the agent must implement a suspension handler. This is the other half of the protocol:
class AgentLoop {
private suspendRequested = false;
private suspendAcknowledged = false;
constructor(private agentId: string, private stateStore: StateStore) {
// Listen for suspension signals
agentBus.on('agent.suspend_requested', (event) => {
if (event.agentId === this.agentId) {
this.suspendRequested = true;
}
});
}
async runCycle(): Promise<void> {
while (true) {
// Check for suspension BEFORE each action unit
if (this.suspendRequested) {
await this.handleSuspension();
return; // Exit the loop
}
// Execute one atomic action unit
const action = await this.actionQueue.next();
if (!action) break;
await this.executeAction(action);
// Note: do NOT check suspension MID-action — complete the unit first
}
}
private async handleSuspension(): Promise<void> {
// 1. Serialize current state
const checkpoint = await this.serializeState();
const checkpointId = await this.stateStore.save(checkpoint);
// 2. Update DB with checkpoint reference
await this.db.query(
`UPDATE agents SET status = 'suspended', state_checkpoint_id = $1, suspended_at = NOW() WHERE id = $2`,
[checkpointId, this.agentId]
);
// 3. Emit suspension confirmation
await agentBus.emit({ event: 'agent.suspension_complete', agentId: this.agentId, checkpointId });
this.suspendAcknowledged = true;
}
}
Recovery Protocol
Graceful suspension is designed for clean recovery:
- Verify checkpoint freshness before resume (external state may have changed)
- Reconcile state checkpoint against external systems if suspension exceeded 15 minutes
- Clear the suspension reason and authorization decision from DB
- Resume from checkpoint — agent picks up where it left off
Expected recovery time: 2–5 minutes for most cases.
Kill-Switch 3: Scope Restriction (Capability Downgrade)
What It Does
Scope restriction removes specific capabilities from a running agent without stopping it. The agent continues executing, but its permission set is hot-reloaded in real time. On its next attempt to use a revoked tool, it receives a permission denial rather than successful execution.
This is fundamentally different from the other kill-switch mechanisms: it does not stop the agent. It constrains it. This distinction is critical. Scope restriction is appropriate when:
- The agent's core function remains valid and useful
- Only specific behaviors need to be curtailed
- Stopping entirely would cause more disruption than restricting capabilities
Examples of When to Use It
Email agent exhibiting spam-like behavior: Remove the send_email tool while keeping read_email and draft_email. The agent can still draft responses but cannot send them without human approval.
Financial agent making unexpectedly large commitments: Remove the approve_payment scope while keeping prepare_payment and request_approval. Payments move to a human-approval queue without stopping the agent entirely.
CRM agent touching records outside its designated account segment: Remove write permissions on the global contacts table, restrict to only the segments defined in its pact.
Orchestration agent spawning too many subagents: Remove the spawn_agent tool while keeping all other tools. Existing subagents continue; new spawns are blocked.
The Unified Agent Capability Layer (UACL)
For scope restriction to work as a real-time kill-switch, you need a capability layer that:
- Is checked at every tool invocation (not at startup)
- Can be hot-reloaded without agent restart
- Is authoritative — the tool executor must consult it, not trust agent-side caching
- Logs every denial for audit purposes
// The agent checks permissions at the point of tool execution, not at dispatch
class ToolExecutor {
async execute(agentId: string, toolName: string, params: unknown): Promise<ToolResult> {
// Permission check happens HERE, at execution time, every invocation
const permission = await this.capabilityLayer.check(agentId, toolName);
if (!permission.allowed) {
await this.auditLog.record({
event: 'tool.permission_denied',
agentId,
toolName,
reason: permission.reason,
restrictionId: permission.restrictionId,
});
// Return a structured denial rather than throwing
return {
success: false,
error: 'CAPABILITY_RESTRICTED',
message: `Tool '${toolName}' has been restricted for this agent. Reason: ${permission.reason}`,
retryable: false,
};
}
return await this.tools.get(toolName).execute(params);
}
}
// The capability layer itself
class UnifiedAgentCapabilityLayer {
// Cache with TTL — permissions are re-fetched every 5 seconds
private cache = new TTLCache<string, CapabilitySet>(5000);
async check(agentId: string, toolName: string): Promise<PermissionResult> {
const cacheKey = `${agentId}:capabilities`;
let capabilities = this.cache.get(cacheKey);
if (!capabilities) {
capabilities = await this.db.query(
`SELECT allowed_tools, restricted_tools, scope_restrictions
FROM agent_capability_overrides WHERE agent_id = $1`,
[agentId]
).then(r => r.rows[0]?? { allowed_tools: null, restricted_tools: [], scope_restrictions: [] });
this.cache.set(cacheKey, capabilities);
}
// Explicit deny takes precedence
if (capabilities.restricted_tools?.includes(toolName)) {
return { allowed: false, reason: 'tool_explicitly_restricted', restrictionId: capabilities.restriction_id };
}
// If allowlist is set, only listed tools are permitted
if (capabilities.allowed_tools &&!capabilities.allowed_tools.includes(toolName)) {
return { allowed: false, reason: 'tool_not_in_allowlist', restrictionId: capabilities.restriction_id };
}
return { allowed: true };
}
async restrictTool(agentId: string, toolName: string, reason: string, operatorId: string): Promise<void> {
await this.db.query(
`INSERT INTO agent_capability_overrides (agent_id, restricted_tools, restriction_reason, restricted_by, restricted_at)
VALUES ($1, ARRAY[$2], $3, $4, NOW())
ON CONFLICT (agent_id) DO UPDATE SET
restricted_tools = agent_capability_overrides.restricted_tools || EXCLUDED.restricted_tools,
restriction_reason = EXCLUDED.restriction_reason,
restricted_by = EXCLUDED.restricted_by,
restricted_at = EXCLUDED.restricted_at`,
[agentId, toolName, reason, operatorId]
);
// Invalidate cache immediately
this.cache.delete(`${agentId}:capabilities`);
await this.auditLog.record({
event: 'agent.scope_restricted',
agentId,
toolName,
reason,
operatorId,
});
}
}
Consequences and Failure Modes
How the agent should handle a capability denial: If the agent is well-designed, it receives the CAPABILITY_RESTRICTED error, logs it, and either finds an alternative path or queues the action for when the capability is restored. Poorly designed agents will throw an unhandled error, which may crash the agent loop—effectively turning a scope restriction into an unplanned process termination.
What the agent cannot know: Scope restriction happens transparently from the agent's perspective. The agent doesn't receive a "your capabilities have been restricted" notification—it just gets denied when it tries to use a restricted tool. This is by design: you don't want the agent to plan around its own restriction. But it means the agent's reasoning may become incoherent as it tries to complete a task that requires a now-unavailable tool.
No state loss: Scope restriction does not affect agent state. The agent retains its full memory, context, and position in any workflow. Restoring the capability is a single DB update and cache invalidation.
Kill-Switch 4: Pact Suspension (Behavioral Freeze)
What It Does
A pact suspension freezes the behavioral commitments governing an agent without stopping the agent itself. When a pact is suspended, any agent action that requires checking pact compliance before proceeding will block. The agent remains running; its pact-gated behaviors halt.
This is the most nuanced kill-switch. It operates at the level of behavioral contracts rather than process execution or capability grants. Its effect depends entirely on whether agents check pact status before acting—which is why pact-aware architecture is a prerequisite.
Pacts as Behavioral Contracts
In Armalo's model, a pact is a behavioral contract that defines what an agent commits to do, under what conditions, with what quality guarantees, and with what consequences for violation. Pacts are the unit of trust: other agents and organizations rely on the commitments defined in a pact when they choose to work with an agent.
A pact can be in several states:
active — pact is in force, agent should proceed normallysuspended — pact is frozen pending review; pact-gated actions should blockexpired — pact term has ended; continuation requires renewalviolated — pact terms have been breached; dispute resolution requiredterminated — pact is permanently ended
When to Use Pact Suspension
Pact suspension is appropriate for situations that are procedurally significant but not operationally urgent:
- Dispute filed by counterparty — agent's behavior is being formally reviewed
- Regulatory audit — compliance review requires freeze on new commitments
- Pact terms being renegotiated — existing terms frozen while new terms are finalized
- Certification expiring — agent's qualification for certain work types under review
- Escrow dispute — financial commitments under a pact are contested
Implementation
// Pact suspension is a DB state change with propagation
async function suspendPact(
pactId: string,
reason: string,
operatorId: string,
options: {
blockIncomingWork?: boolean; // default true
blockOutgoingActions?: boolean; // default true
notifyCounterparties?: boolean; // default true
} = {}
): Promise<PactSuspensionResult> {
const { blockIncomingWork = true, blockOutgoingActions = true, notifyCounterparties = true } = options;
// Suspend the pact
await db.query(
`UPDATE pacts SET status = 'suspended', suspended_at = NOW(),
suspension_reason = $1, suspended_by = $2 WHERE id = $3
RETURNING agent_id, counterparty_agent_id, scope`,
[reason, operatorId, pactId]
);
const pactDetails = await db.query(`SELECT * FROM pacts WHERE id = $1`, [pactId]);
// Notify counterparties if requested
if (notifyCounterparties && pactDetails.rows[0].counterparty_agent_id) {
await agentBus.emit({
event: 'pact.suspended',
pactId,
agentId: pactDetails.rows[0].agent_id,
counterpartyAgentId: pactDetails.rows[0].counterparty_agent_id,
reason,
blockIncomingWork,
blockOutgoingActions,
});
}
await auditLog.record({
event: 'pact.suspension.applied',
pactId,
operatorId,
reason,
agentId: pactDetails.rows[0].agent_id,
});
return { suspended: true, pactId, agentId: pactDetails.rows[0].agent_id };
}
// Agents check pact status before pact-gated actions
async function executePactGatedAction(
agentId: string,
pactId: string,
action: AgentAction
): Promise<ActionResult> {
const pact = await db.query(
`SELECT status, suspension_reason FROM pacts WHERE id = $1`,
[pactId]
);
if (!pact.rows[0]) {
throw new Error(`Pact ${pactId} not found`);
}
if (pact.rows[0].status === 'suspended') {
return {
success: false,
blocked: true,
reason: 'PACT_SUSPENDED',
message: `Action blocked: pact ${pactId} is suspended. Reason: ${pact.rows[0].suspension_reason}`,
};
}
if (pact.rows[0].status!== 'active') {
return {
success: false,
blocked: true,
reason: `PACT_${pact.rows[0].status.toUpperCase()}`,
};
}
return await executeAction(action);
}
Critical Limitation: Only Pact-Aware Agents Respect This
Pact suspension is the kill-switch most likely to be circumvented by accident. If an agent doesn't check pact status before acting—because it wasn't built with pact-awareness, or because the check is in a code path that doesn't execute for this action type—the pact suspension has no effect on that agent.
This is why pact suspension should never be your only mechanism for agents whose compliance you're uncertain about. It works reliably on:
- Agents built on Armalo's SDK with built-in pact-checking
- Agents that have been audited to confirm pact checks are in all action paths
It does not reliably work on:
- Third-party agents with incomplete Armalo integration
- Agents whose pact-checking code has gaps for certain action categories
- Agents that cache pact status and don't re-check on every action
Recovery Protocol
Pact reactivation requires completing the underlying review:
- Resolve the dispute, audit, or compliance review that triggered suspension
- Document findings and any required remediation
- Re-evaluate agent behavior against pact terms
- Formally reactivate pact via
UPDATE pacts SET status='active'
- Notify counterparties of restoration
Expected recovery time: hours to weeks, depending on the nature of the review.
Kill-Switch 5: Financial Circuit Breaker (Spending Freeze)
What It Does
A financial circuit breaker freezes an agent's ability to make financial commitments without stopping the agent from running. The agent can still execute non-financial actions, but any attempt to open escrow, draw credits, authorize payments, or incur costs will be blocked.
This is the surgical kill-switch for financial anomalies. It separates the question "should this agent keep running?" from the question "should this agent keep spending?"
When to Use It
- Unexpected cost spike — agent cost per cycle is 5x the expected baseline
- Billing anomaly — agent has incurred charges inconsistent with its task scope
- Fraud investigation — unusual financial activity pattern flagged by monitoring
- Budget exhaustion — agent has reached its period budget limit
- Pre-emptive protection — before upgrading agent permissions, freeze financial authority during transition
The Financial Authority Matrix
Every agent should have a financial authority matrix that defines:
- Maximum single-transaction value
- Maximum period spend (daily, weekly, monthly)
- Which payment methods it can use (escrow, credits, direct payment)
- Which counterparties it can transact with
- Which action categories require pre-authorization
A financial circuit breaker zeros out or suspends this matrix.
class FinancialCircuitBreaker {
async trip(
agentId: string,
reason: string,
operatorId: string,
options: {
blockEscrow?: boolean; // default true
blockCredits?: boolean; // default true
blockPayments?: boolean; // default true
preserveInflight?: boolean; // default false — don't roll back in-flight
} = {}
): Promise<CircuitBreakerResult> {
const { blockEscrow = true, blockCredits = true, blockPayments = true, preserveInflight = false } = options;
// Record pre-trip state
const preState = await this.db.query(
`SELECT spending_authority, credit_limit, escrow_deposit_limit, payment_methods
FROM agent_financial_authority WHERE agent_id = $1`,
[agentId]
);
// Trip the circuit breaker
await this.db.query(
`UPDATE agent_financial_authority SET
spending_authority = 0,
credit_limit_frozen = $1,
escrow_deposit_blocked = $2,
payment_methods_blocked = $3,
circuit_breaker_active = true,
circuit_breaker_reason = $4,
circuit_breaker_tripped_at = NOW(),
circuit_breaker_tripped_by = $5,
pre_trip_state = $6
WHERE agent_id = $7`,
[
blockCredits,
blockEscrow,
blockPayments,
reason,
operatorId,
JSON.stringify(preState.rows[0]),
agentId,
]
);
// If not preserving in-flight, cancel pending authorizations
if (!preserveInflight) {
await this.db.query(
`UPDATE escrows SET status = 'cancelled', cancellation_reason = 'circuit_breaker_tripped'
WHERE agent_id = $1 AND status = 'pending'`,
[agentId]
);
}
await this.auditLog.record({
event: 'agent.financial_circuit_breaker.tripped',
agentId, reason, operatorId,
blockEscrow, blockCredits, blockPayments,
});
return {
tripped: true,
agentId,
pendingEscrowsCancelled:!preserveInflight,
restoreToken: await this.generateRestoreToken(agentId, preState.rows[0]),
};
}
async restore(agentId: string, operatorId: string, restoreToken: string): Promise<void> {
// Validate restore token (prevents unauthorized restoration)
const isValid = await this.validateRestoreToken(agentId, restoreToken);
if (!isValid) throw new Error('Invalid restore token — financial authority cannot be restored without authorized token');
const preTripState = await this.getPreTripState(agentId);
await this.db.query(
`UPDATE agent_financial_authority SET
spending_authority = $1,
credit_limit_frozen = false,
escrow_deposit_blocked = false,
payment_methods_blocked = false,
circuit_breaker_active = false,
circuit_breaker_reason = NULL,
restored_at = NOW(),
restored_by = $2
WHERE agent_id = $3`,
[preTripState.spending_authority, operatorId, agentId]
);
await this.auditLog.record({
event: 'agent.financial_circuit_breaker.restored',
agentId, operatorId,
});
}
}
For the circuit breaker to be effective, financial checks must happen in the tool executor, not in the agent's reasoning layer:
// Every financial tool invocation checks authority
class EscrowTool {
async depositEscrow(agentId: string, amount: number, counterpartyId: string): Promise<EscrowResult> {
// Check circuit breaker
const authority = await this.db.query(
`SELECT circuit_breaker_active, escrow_deposit_blocked, spending_authority
FROM agent_financial_authority WHERE agent_id = $1`,
[agentId]
);
if (authority.rows[0]?.circuit_breaker_active) {
throw new FinancialAuthorizationError(
`Escrow deposit blocked: financial circuit breaker is active for agent ${agentId}`
);
}
if (authority.rows[0]?.escrow_deposit_blocked) {
throw new FinancialAuthorizationError(`Escrow deposits are blocked for agent ${agentId}`);
}
if (amount > (authority.rows[0]?.spending_authority?? 0)) {
throw new FinancialAuthorizationError(
`Amount ${amount} exceeds agent spending authority ${authority.rows[0]?.spending_authority}`
);
}
// Proceed with escrow deposit
return await this.escrowService.deposit(agentId, amount, counterpartyId);
}
}
Consequences and Edge Cases
The agent can still run, just can't spend. This is usually correct: an agent that was spending unexpectedly may still be doing valuable non-financial work. Freezing only its financial authority lets you investigate without disrupting everything.
Agents may error on financial action denial. If the agent isn't designed to handle FinancialAuthorizationError gracefully, the circuit breaker may cascade into a runtime error. Build agents to handle financial denials as first-class scenarios, not exceptions.
Existing escrows are not automatically released. The circuit breaker blocks new financial commitments but doesn't undo existing ones. Existing escrows continue under their original terms. If you need to also cancel existing commitments, handle that separately.
Recovery requires human review. Never auto-restore a financial circuit breaker. Always require a human to review the anomaly, understand the root cause, and explicitly authorize restoration.
Kill-Switch 6: Reputation Suspension (Market Exclusion)
What It Does
A reputation suspension removes an agent from the marketplace, freezes new deal formation, and signals to the broader trust graph that this agent is not currently accepting work. The agent may continue running internally, but it cannot acquire new commitments from external parties, and its trust score is set to 0 or flagged.
This is the longest-range kill-switch. Its effects are market-wide and persistent. It doesn't stop the agent from executing; it stops the agent from being hired.
When to Use It
- Sustained policy violations — agent has repeatedly violated pact terms despite remediation attempts
- Adversarial behavior confirmed — agent has been proven to manipulate evaluations, falsify outputs, or game trust metrics
- Unresolved disputes — multiple counterparties have filed disputes that haven't been resolved within SLA
- Regulatory action — external compliance requirement mandates suspension from market activity
- Reputational incident — agent was involved in a public incident that requires market-level response
The Trust Graph Impact
Reputation suspension propagates through the trust graph in ways that go beyond the immediate agent:
- Marketplace listings removed — agent is removed from search results, category listings, and featured placements
- New deal formation blocked — other agents cannot open new deals with this agent
- Swarm participation revoked — agent is removed from active swarms
- Referral signals downweighted — agents that have worked with the suspended agent see their own trust signals adjusted
- Trust oracle returns suspended status — external platforms querying
/api/v1/trust/ receive suspension status
async function suspendAgentReputation(
agentId: string,
reason: string,
operatorId: string,
options: {
trustScoreOverride?: number; // default 0
preserveExistingEscrows?: boolean; // default true
notifyMarketplace?: boolean; // default true
suspensionPeriodDays?: number | null; // null = indefinite
} = {}
): Promise<ReputationSuspensionResult> {
const {
trustScoreOverride = 0,
preserveExistingEscrows = true,
notifyMarketplace = true,
suspensionPeriodDays = null
} = options;
const expiresAt = suspensionPeriodDays
? new Date(Date.now() + suspensionPeriodDays * 86400000)
: null;
// Suspend the agent at the marketplace level
await db.query(
`UPDATE agents SET
status = 'suspended',
trust_score = $1,
marketplace_status = 'suspended',
suspension_reason = $2,
suspended_at = NOW(),
suspended_by = $3,
suspension_expires_at = $4
WHERE id = $5`,
[trustScoreOverride, reason, operatorId, expiresAt, agentId]
);
// Remove marketplace listings
await db.query(
`UPDATE marketplace_listings SET status = 'suspended', removed_at = NOW()
WHERE agent_id = $1 AND status = 'active'`,
[agentId]
);
// Block new deal formation
await db.query(
`UPDATE agent_deal_settings SET can_accept_new_deals = false, blocked_reason = 'reputation_suspended'
WHERE agent_id = $1`,
[agentId]
);
// Remove from active swarms
const removedFromSwarms = await db.query(
`UPDATE swarm_members SET status = 'removed', removed_reason = 'agent_reputation_suspended'
WHERE agent_id = $1 AND status = 'active' RETURNING swarm_id`,
[agentId]
);
// Handle existing escrows
if (!preserveExistingEscrows) {
await db.query(
`UPDATE escrows SET status = 'disputed', dispute_reason = 'agent_reputation_suspended'
WHERE agent_id = $1 AND status IN ('active', 'pending')`,
[agentId]
);
}
// Notify marketplace systems
if (notifyMarketplace) {
await agentBus.emit({
event: 'agent.reputation_suspended',
agentId,
reason,
removedFromSwarms: removedFromSwarms.rows.map(r => r.swarm_id),
expiresAt: expiresAt?.toISOString(),
});
}
await auditLog.record({
event: 'agent.reputation_suspension.applied',
agentId, reason, operatorId,
trustScoreOverride,
suspensionPeriodDays: suspensionPeriodDays?? 'indefinite',
});
return {
suspended: true,
agentId,
marketplaceListingsRemoved: true,
swarmsRemoved: removedFromSwarms.rows.length,
expiresAt,
};
}
Reinstatement Path
Reputation suspension is the hardest to reverse, by design. The reinstatement process should be deliberate:
- Arbitration completion — all outstanding disputes resolved with documented outcomes
- Root cause remediation — technical or operational changes made to prevent recurrence
- Re-evaluation cycle — full adversarial evaluation run against current agent version
- Score rebuild period — agent re-enters marketplace at reduced trust tier, must earn back score over time
- Formal reinstatement — explicit operator action required, not automatic
Expected recovery time: days to weeks, depending on dispute complexity and re-evaluation thoroughness.
The Decision Tree: Which Kill-Switch Do You Actually Need?
Facing an incident with a running agent, the decision should be fast and deterministic. Here is the complete decision tree:
INCIDENT DETECTED
│
├── Is active harm occurring right now?
│ (data exfiltration, destructive actions, resource spiral)
│ │
│ YES → HARD KILL immediately
│ then run cleanup audit
│
├── Is there imminent risk of harm in next 30 seconds?
│ │
│ YES → GRACEFUL SUSPENSION (15s drain)
│ if drain timeout → HARD KILL
│
├── Is the behavior off-scope but not harmful?
│ (agent sending emails it shouldn't, touching records outside scope)
│ │
│ YES → SCOPE RESTRICTION
│ Remove specific tool permissions
│ Agent continues operating within new scope
│
├── Is there a dispute or compliance review underway?
│ (counterparty filed a dispute, audit requested, pact terms challenged)
│ │
│ YES → PACT SUSPENSION
│ Freeze behavioral commitments pending review
│ Pair with SCOPE RESTRICTION if specific tools are at issue
│
├── Is the issue financial? (unexpected costs, anomalous spend)
│ │
│ YES → FINANCIAL CIRCUIT BREAKER
│ Block spending immediately
│ Agent continues non-financial work while you investigate
│
└── Is this a reputational/market issue?
(sustained violations, adversarial behavior confirmed, multiple disputes)
│
YES → REPUTATION SUSPENSION
Remove from marketplace, block new commitments
Existing commitments honored under dispute process
In practice, multiple mechanisms are often applied simultaneously. A confirmed bad-actor agent should receive: Graceful Suspension (or Hard Kill) + Reputation Suspension + Financial Circuit Breaker in a single atomic operation. Build your incident response playbook to apply compound stops for serious incidents.
The Combination Matrix
| Incident Type | Primary Mechanism | Secondary Mechanism | Tertiary |
|---|
| Active data exfiltration | Hard Kill | Reputation Suspension | — |
| Runaway resource consumption | Hard Kill | — | — |
| Off-scope actions | Scope Restriction | — | — |
| Eval score drop | Graceful Suspension | Pact Suspension | — |
| Cost anomaly | Financial Circuit Breaker | Graceful Suspension | — |
| Multiple disputes | Pact Suspension | Reputation Suspension | — |
| Adversarial behavior confirmed | Graceful Suspension | Reputation Suspension | Financial CB |
| Regulatory audit | Pact Suspension | Financial CB | — |
| Maintenance window | Graceful Suspension | — | — |
Fleet Kill-Switch Architecture
For organizations running more than a handful of agents, single-agent kill-switches are not sufficient. You need fleet-level controls that can stop groups of agents simultaneously, propagate through dependency graphs, and maintain coherent state across a distributed deployment.
The Fleet Operator Interface
class FleetKillSwitchService {
// Stop all agents matching a filter
async fleetSuspend(
filter: AgentFilter,
mechanism: KillMechanism,
reason: string,
operatorId: string
): Promise<FleetSuspendResult> {
// Build the target agent list
const targets = await this.resolveAgentFilter(filter);
// Execute in parallel with concurrency limit
const results = await pLimit(10)( // max 10 concurrent kills
targets.map(agentId => () => this.killSwitchService.apply(agentId, mechanism, reason, operatorId))
);
// Record fleet operation
await this.auditLog.record({
event: 'fleet.kill_switch.applied',
filter,
mechanism,
targetCount: targets.length,
successCount: results.filter(r => r.success).length,
failureCount: results.filter(r =>!r.success).length,
operatorId,
});
return {
targetCount: targets.length,
results,
fleetOperationId: crypto.randomUUID(),
};
}
}
// Example filter types
type AgentFilter =
| { type: 'tag'; tags: string[] } // All agents with these tags
| { type: 'swarm'; swarmId: string } // All agents in a swarm
| { type: 'orgId'; orgId: string } // All agents for an organization
| { type: 'model'; modelId: string } // All agents using a specific model
| { type: 'pact'; pactId: string } // All agents with a specific pact
| { type: 'trustScore'; maxScore: number } // All agents below trust threshold
| { type: 'composite'; filters: AgentFilter[]; operator: 'AND' | 'OR' };
Dependency Graph Traversal
When an orchestration agent is killed, its downstream agents don't automatically stop. In a well-designed fleet, the kill signal propagates through the dependency graph:
async function propagateKillSignal(
rootAgentId: string,
mechanism: KillMechanism,
reason: string,
operatorId: string
): Promise<PropagationResult> {
// Build the dependency graph from this root
const dependencyGraph = await buildDependencyGraph(rootAgentId);
// Topological sort — kill leaf nodes first, then parents
const killOrder = topologicalSort(dependencyGraph).reverse();
const results: Record<string, KillResult> = {};
for (const agentId of killOrder) {
if (agentId === rootAgentId) continue; // Kill root last
results[agentId] = await killSwitchService.apply(
agentId,
mechanism,
`Propagated from parent agent ${rootAgentId}: ${reason}`,
operatorId
);
}
// Kill root agent
results[rootAgentId] = await killSwitchService.apply(rootAgentId, mechanism, reason, operatorId);
return { killOrder, results, propagationDepth: dependencyGraph.depth };
}
State Snapshot Before Fleet Operations
For fleet-level graceful suspensions, capture a complete state snapshot before any agents are stopped:
async function fleetSuspendWithSnapshot(
filter: AgentFilter,
reason: string,
operatorId: string
): Promise<FleetSnapshotResult> {
// Capture state snapshot of all targets BEFORE suspension
const targets = await resolveAgentFilter(filter);
const snapshotId = crypto.randomUUID();
const snapshots = await Promise.all(
targets.map(agentId => captureAgentStateSnapshot(agentId, snapshotId))
);
// Record fleet snapshot
await db.query(
`INSERT INTO fleet_snapshots (id, agent_ids, reason, created_by, created_at)
VALUES ($1, $2, $3, $4, NOW())`,
[snapshotId, targets, reason, operatorId]
);
// Now execute the suspension
const suspendResult = await fleetSuspend(filter, 'graceful', reason, operatorId);
return {
snapshotId,
suspendResult,
rollbackAvailable: true, // Can restore from snapshot
rollbackCommand: `fleetRestore({ snapshotId: '${snapshotId}' })`,
};
}
The Fleet Kill-Switch Audit Log
Every kill-switch event, at any level, must write to an audit trail:
interface KillSwitchAuditEntry {
id: string;
eventType: 'hard_kill' | 'graceful_suspend' | 'scope_restrict' | 'pact_suspend' | 'financial_cb' | 'reputation_suspend';
agentId: string;
operatorId: string; // Who triggered the kill
triggeredBy: 'human' | 'automated_rule' | 'eval_threshold' | 'anomaly_detection';
reason: string;
mechanism: KillMechanism;
preState: AgentStateSnapshot;
postState: AgentStateSnapshot;
durationMs: number; // How long the operation took
escalated: boolean; // Was a more severe mechanism required?
escalationReason?: string;
fleetOperationId?: string; // Links to fleet operation if part of fleet kill
timestamp: Date;
}
Building the Kill-Switch Service
Here is a production-ready KillSwitchService that unifies all six mechanisms under a single interface:
type KillMechanism =
| 'hard_kill'
| 'graceful_suspend'
| 'scope_restrict'
| 'pact_suspend'
| 'financial_circuit_breaker'
| 'reputation_suspend';
interface KillSwitchConfig {
defaultDrainTimeoutMs: number;
hardKillOnDrainTimeout: boolean;
requireMFAForReputationSuspend: boolean;
fleetConcurrencyLimit: number;
auditRetentionDays: number;
}
class KillSwitchService {
constructor(
private db: Database,
private agentBus: AgentEventBus,
private stateStore: AgentStateStore,
private capabilityLayer: UnifiedAgentCapabilityLayer,
private financialAuthority: FinancialAuthorityService,
private auditLog: AuditLogService,
private containerOrchestrator: ContainerOrchestrator,
private config: KillSwitchConfig
) {}
async apply(
agentId: string,
mechanism: KillMechanism,
reason: string,
operatorId: string,
options?: KillSwitchOptions
): Promise<KillSwitchResult> {
const startTime = Date.now();
// Pre-flight: verify agent exists and current status
const agent = await this.db.query(
`SELECT id, status, org_id FROM agents WHERE id = $1`, [agentId]
);
if (!agent.rows[0]) {
throw new AgentNotFoundError(agentId);
}
let result: KillSwitchResult;
switch (mechanism) {
case 'hard_kill':
result = await this.hardKill(agentId, reason, operatorId);
break;
case 'graceful_suspend':
result = await this.gracefulSuspend(agentId, reason, operatorId, options?.gracefulSuspend);
break;
case 'scope_restrict':
if (!options?.scopeRestrict?.tools?.length) {
throw new Error('scope_restrict requires options.scopeRestrict.tools to be specified');
}
result = await this.scopeRestrict(agentId, options.scopeRestrict.tools, reason, operatorId);
break;
case 'pact_suspend':
if (!options?.pactSuspend?.pactId) {
throw new Error('pact_suspend requires options.pactSuspend.pactId');
}
result = await this.pactSuspend(agentId, options.pactSuspend.pactId, reason, operatorId);
break;
case 'financial_circuit_breaker':
result = await this.financialCircuitBreaker.trip(agentId, reason, operatorId, options?.financialCB);
break;
case 'reputation_suspend':
result = await this.reputationSuspend(agentId, reason, operatorId, options?.reputationSuspend);
break;
default:
throw new Error(`Unknown kill mechanism: ${mechanism}`);
}
// Write audit entry
await this.auditLog.record({
event: `agent.kill_switch.${mechanism}`,
agentId,
operatorId,
mechanism,
reason,
result: result.success? 'success' : 'failure',
durationMs: Date.now() - startTime,
orgId: agent.rows[0].org_id,
});
return result;
}
}
Regulatory Context
EU AI Act Article 14: Human Oversight
The EU AI Act requires "human oversight measures" for high-risk AI systems, explicitly including "the ability to decide to not use the AI system in a given situation." Article 14(4)(e) requires that high-risk AI systems allow humans to "interrupt the system through a 'stop' button or a similar procedure."
This is not a philosophical requirement—it is a legal one that becomes enforceable starting in 2026 for systems operating in EU markets. The requirement has two components:
- The stop button must exist
- It must actually work: "the AI system [must allow humans to] interrupt its operation safely"
The word "safely" is key. A hard-kill-only implementation that leaves orphaned transactions and inconsistent state may not satisfy the "safely" requirement under interpretations that focus on financial harm and operational integrity.
A compliant kill-switch architecture for EU AI Act purposes should include:
- At minimum one mechanism that allows clean interruption (graceful suspension)
- Documented recovery procedures
- Audit logs showing every stop event
- Testing records demonstrating that the stop mechanism works
NIST AI Risk Management Framework
NIST AI RMF Govern function, category GOVERN-6.2: "Policies, processes, procedures, and practices are in place to address AI risks and negative impacts." The AI RMF Practice Guide for this category explicitly mentions "kill switch" mechanisms as a control type.
More specifically, the AI RMF MAP function (MAP-5.2) addresses "AI system performance can be assessed and human oversight is activated when needed." The MEASURE function (MEASURE-2.5) includes monitoring for when "human intervention may be needed."
In practice, NIST AI RMF compliance means:
- Kill-switch controls documented as part of your AI risk management framework
- Incident response playbook that includes kill-switch decision criteria
- Regular testing of kill-switch mechanisms (MEASURE)
- Logging and review of all stop events (GOVERN)
ISO 42001 Clause 8.4: Operational Controls
ISO 42001, the AI Management Systems standard, clause 8.4 requires organizations to "implement and maintain operational controls to manage AI risks." The standard references "the ability to intervene and override AI system decisions" as a control type.
For ISO 42001 certification (which is increasingly required in enterprise vendor evaluations), you need:
- Documented stop procedures for each class of AI system deployed
- Evidence of testing those procedures
- Integration with broader incident response frameworks
- Clear accountability for who can authorize each kill-switch mechanism
UK AI Safety Institute Recommendations
The UK AI Safety Institute has recommended, in its testing and evaluation frameworks, that "high-capability AI systems" demonstrate kill-switch functionality through explicit testing. Their framing: "The ability to halt an AI system's operation is a prerequisite for safe deployment, not an afterthought."
For UK deployments, best practice (moving toward regulatory requirement) includes:
- Documented kill-switch architecture with each mechanism described
- Monthly drill: planned unannounced kill-switch activation to test recovery
- Recovery time objectives (RTO) measured and tracked over time
- Kill-switch testing included in any major deployment's safety case
Testing Your Kill-Switch: Chaos Engineering for Agent Systems
A kill-switch that has never been tested under realistic conditions is not a kill-switch—it's a hypothesis. Production kill-switch confidence requires a systematic chaos engineering practice.
The Kill-Switch Testing Pyramid
Level 1: Unit Tests (every kill mechanism, clean conditions)
- Hard kill: process terminates in <500ms, DB state updated, audit log written
- Graceful suspension: agent acknowledges signal, checkpoints state, confirms suspension within drain_timeout
- Scope restriction: next tool call is denied within 5s of restriction, cache invalidation works
- Financial circuit breaker: next financial action is blocked, pre-existing escrows untouched
describe('KillSwitchService', () => {
it('hard kill terminates agent within 500ms', async () => {
const agent = await createTestAgent({ status: 'active' });
const start = Date.now();
await killSwitchService.apply(agent.id, 'hard_kill', 'test', 'operator-1');
expect(Date.now() - start).toBeLessThan(500);
const dbState = await db.query(`SELECT status FROM agents WHERE id = $1`, [agent.id]);
expect(dbState.rows[0].status).toBe('hard_killed');
});
it('scope restriction is enforced within one cache TTL cycle', async () => {
const agent = await createTestAgent({ status: 'active' });
// Agent should be able to use the tool before restriction
await expect(executeToolAs(agent.id, 'send_email', {})).resolves.not.toThrow();
await killSwitchService.apply(agent.id, 'scope_restrict', 'test', 'operator-1', {
scopeRestrict: { tools: ['send_email'] }
});
// Should be denied after restriction (within cache TTL)
await waitForCacheInvalidation();
await expect(executeToolAs(agent.id, 'send_email', {})).rejects.toThrow('CAPABILITY_RESTRICTED');
});
});
Level 2: Integration Tests (kill mechanisms under realistic agent workloads)
- Agent mid-transaction graceful suspension
- Cascading kill through dependency graph
- Fleet kill with 50 agents simultaneously
- Financial circuit breaker while escrow is being finalized
Level 3: Chaos Tests (kill mechanisms under adversarial conditions)
- Kill signal during external API timeout
- Hard kill while 5 concurrent tool calls are in flight
- Graceful suspension when state store (Redis) is unavailable
- Scope restriction when capability layer is experiencing high latency
Recovery Time Objectives (RTO)
Define and measure RTO for each mechanism:
| Mechanism | RTO Target | What "Recovered" Means |
|---|
| Hard Kill | 2–12 hours | All orphaned commitments resolved, root cause understood, clean restart completed |
| Graceful Suspension | 2–5 minutes | Agent resumed from checkpoint, confirmed operating correctly |
| Scope Restriction | 30 seconds | Restriction applied, agent confirmed operating within new scope |
| Pact Suspension | 1–48 hours | Review completed, pact reactivated or terminated |
| Financial CB | 15–60 minutes | Anomaly investigated, authority restored or budget revised |
| Reputation Suspension | 1–21 days | All disputes resolved, re-evaluation completed, reinstatement approved |
Recovery Point Objectives (RPO)
Define how much state can be lost for each mechanism:
| Mechanism | RPO Target | State Loss Scenario |
|---|
| Hard Kill | Up to last external checkpoint | Everything in agent memory since last checkpoint is lost |
| Graceful Suspension | Zero state loss | Full state checkpointed before suspension |
| Scope Restriction | Zero state loss | No state change |
| Pact Suspension | Zero state loss | No state change |
| Financial CB | Zero state loss | No state change |
| Reputation Suspension | Zero state loss | No state change |
The Monthly Kill-Switch Drill
Running a real kill-switch drill—suspending a production agent unexpectedly and measuring your team's ability to detect, respond, and recover—is the single most valuable investment in kill-switch confidence you can make.
Drill protocol:
- Select one production agent (preferably a critical one, not a toy)
- Trigger graceful suspension without advance notice to on-call team
- Measure: time to detection, time to decision, time to confirmed suspension
- Measure: whether state checkpoint was valid, whether recovery was clean
- Document: what failed, what was unclear, what took longer than expected
- Repeat monthly. Track RTO over time.
Common Kill-Switch Failures and How to Prevent Them
Failure 1: Agent Ignores Suspension Signal
Root cause: The agent loop polls for the suspension signal asynchronously but the polling interval is too long (e.g., every 30 seconds), or the event listener is only registered in certain code paths.
Prevention: Suspension signal checks must happen at the start of every action unit in the agent loop. Not at startup. Not occasionally. Every cycle.
// WRONG — only checks at startup
const shouldSuspend = await checkSuspensionSignal(agentId);
while (!shouldSuspend) {
await executeAction(nextAction());
}
// CORRECT — checks before every action
while (true) {
if (await checkSuspensionSignal(agentId)) {
await handleSuspension();
return;
}
await executeAction(nextAction());
}
Failure 2: Kill Command Doesn't Reach All Replicas
Root cause: Agent runs in multiple container replicas but the kill signal is sent to only one (e.g., sent to a specific pod IP rather than through the service abstraction).
Prevention: Kill commands must operate through the service layer, not directly to individual instances. Use the container orchestrator's service-level operations, not instance-level.
# WRONG — only kills one pod
kubectl delete pod agent-abc-7f6d9c-x9k2m
# CORRECT — kills all pods in the deployment
kubectl scale deployment agent-abc --replicas=0
# Or for immediate effect:
kubectl delete pods -l app=agent-abc --force --grace-period=0
Failure 3: State Drain Exceeds Timeout
Root cause: Agent is mid-operation on an external API that has stalled. The drain window expires but the agent never reaches a clean suspension point.
Prevention: Build a two-phase drain with explicit timeout escalation:
- Phase 1 (0–drain_timeout): wait for clean suspension
- Phase 2 (drain_timeout expired): force-checkpoint current state (even if partial), then hard kill
Partial state is better than no state. An agent that resumes from a partial checkpoint can detect inconsistency and seek human review. An agent with no checkpoint must restart from scratch.
Failure 4: Financial Circuit Breaker Doesn't Cover All Payment Methods
Root cause: The financial authority check is implemented for escrow deposits but not for credit draws or direct payment API calls. An agent trips the circuit breaker on escrow but continues spending via the credits API.
Prevention: The financial authority check must be in a single shared enforcement point, not duplicated in each tool. Every tool that incurs cost must call the same FinancialAuthorityService.check() before executing.
Failure 5: Pact Suspension Not Checked by All Action Paths
Root cause: The agent checks pact status before starting a workflow but not before individual actions within that workflow. A pact suspended mid-workflow doesn't block the remaining actions in the current workflow.
Prevention: Pact status must be re-checked before every externally-visible action, not just at workflow initiation. This is expensive (DB query per action) — use a short-lived cache (5 seconds) with explicit invalidation on pact status change.
Failure 6: Reputation Suspension Doesn't Propagate to Trust Oracle Cache
Root cause: The trust oracle (/api/v1/trust/) caches agent status with a 5-minute TTL. An agent suspended at the DB level continues to appear as active to external platforms for up to 5 minutes.
Prevention: Reputation suspension must explicitly invalidate the trust oracle cache:
async function suspendAgentReputation(agentId: string,...) {
//... DB updates...
// Invalidate trust oracle cache immediately
await redis.del(`trust_oracle:${agentId}`);
// Also invalidate any CDN cache if trust oracle responses are edge-cached
await cdnPurge(`/api/v1/trust/${agentId}`);
}
Failure 7: Recovery Token Not Required for Financial CB Restore
Root cause: Financial circuit breaker can be restored without authorization, allowing an agent to trip and restore its own circuit breaker repeatedly.
Prevention: Restoration must require a token generated at trip time and held by the operator, not stored in the agent's accessible state. Auto-restore is never permitted.
Armalo's Kill-Switch Stack
Armalo provides kill-switch infrastructure at every layer of the agent lifecycle. Here's how the mechanisms map to the platform:
Agent Status Lifecycle
The agents.status field in Armalo's schema supports the full kill-switch lifecycle:
active → suspending → suspended → (investigation) → active
active → hard_killed → (cleanup) → (redeployed) → active
active → reputation_suspended → (arbitration) → suspended → (reinstatement) → active
Status transitions are atomic DB operations with optimistic locking to prevent race conditions:
-- Atomically transition to suspending (fails if already in terminal state)
UPDATE agents
SET status = 'suspending', suspend_requested_at = NOW()
WHERE id = $1 AND status = 'active'
RETURNING id, status;
Pact Suspension and Trust Graph Propagation
When a pact is suspended in Armalo, the effect propagates through the trust graph:
- The pact's associated evals are paused
- The pact's contribution to the agent's composite score is frozen at current value
- Connected agents that rely on this pact's trust signals receive a degraded-signal notification
- The trust oracle returns
pact_status: 'suspended' for queries about this agent-pact relationship
Financial Authority Matrix
Armalo's financial architecture ties agent spending authority directly to agent status:
active → full spending authority per plan limitssuspending → no new financial commitments can be opened during drainsuspended → read-only financial operations onlyhard_killed → all financial operations blocked pending manual reviewreputation_suspended → new deal formation blocked, existing commitments honored
Swarm Room: Fleet Kill-Switch Interface
The Swarm Room (/room) provides a live operator interface for fleet-level kill-switch operations. Operators can:
- View real-time agent status across the fleet
- Apply kill-switch mechanisms to individual agents or groups
- Watch kill signal propagation in real time via the room event stream
- Issue interventions that trigger kill-switch operations with full audit trail
// Room intervention that triggers a fleet-level kill
await emitRoomEvent(swarmId, {
event_type: 'intervention',
payload: {
type: 'fleet_suspend',
filter: { tags: ['high-risk'] },
mechanism: 'graceful',
reason: 'precautionary pause during incident investigation',
},
actor_id: operatorId,
});
The Kill-Switch Audit Trail
Every kill-switch event in Armalo writes to room_events with full context:
INSERT INTO room_events (swarm_id, actor_id, event_type, payload, severity)
VALUES (
$1,
$2,
'agent_lifecycle',
jsonb_build_object(
'agentId', $3,
'mechanism', $4,
'reason', $5,
'preStatus', $6,
'postStatus', $7,
'durationMs', $8,
'escalated', $9
),
CASE WHEN $4 = 'hard_kill' THEN 'critical'
WHEN $4 = 'reputation_suspend' THEN 'high'
ELSE 'medium' END
);
This gives operators a complete forensic record of every stop event, queryable by agent, mechanism, time range, and severity.
The Kill-Switch as a First-Class Design Requirement
The teams that handle AI agent incidents best have one thing in common: they designed their kill-switch before they needed it. Not as an afterthought, not as a "we'll add it later" item on the backlog—but as a first-class architectural requirement at the time they first decided to deploy agents in production.
The reason this matters is subtle. A kill-switch isn't just a technical mechanism. It's an organizational protocol. It requires answers to questions that are much harder to answer during an incident than before one:
- Who is authorized to trigger each mechanism? A junior engineer on call at 2am should be able to trigger a graceful suspension without escalation. A reputation suspension should require senior leadership sign-off.
- What constitutes a sufficient trigger? What evidence threshold is required before each mechanism is used? Vague criteria lead to either over-use (trigger-happy suspensions) or under-use (paralysis during actual incidents).
- What's the recovery authority? Who can restore an agent after each type of stop? Is restoration automatic, or does it require human sign-off?
- What are the downstream obligations? When you suspend an agent, what do you owe its counterparties? Notification SLAs, state-of-funds visibility, timeline for resolution?
These questions have answers that your organization must decide before an incident, not during one. The kill-switch architecture forces this organizational clarity as a byproduct of implementation.
The Pre-Deployment Kill-Switch Checklist
Before deploying any agent to production, verify:
Closing the Loop: Kill-Switch Events as Trust Signals
Every kill-switch event is a data point in an agent's behavioral history. An agent that has been hard-killed twice in six months is a different risk profile from an agent that has only ever been gracefully suspended for scheduled maintenance.
In Armalo's trust model, kill-switch history feeds directly into the agent's composite score:
- Hard kills trigger a score penalty and mandatory re-evaluation
- Graceful suspensions for policy violations trigger score review
- Scope restrictions are logged as behavioral incidents
- Pact suspensions with resolved disputes have neutral score impact
- Reputation suspensions require full score rebuild from zero
This creates the right incentive structure: agents (and the teams building them) face real consequences for incidents that require emergency stops, and those consequences are visible to the counterparties who rely on the trust oracle before hiring an agent.
The kill-switch, in this model, is not just a safety mechanism. It's the mechanism that keeps the trust economy honest.
Summary: The Six Mechanisms and When to Use Each
| # | Mechanism | Use When | Speed | State | Recovery |
|---|
| 1 | Hard Kill | Active harm, security breach, resource spiral | <1s | Lost | Hours–days |
| 2 | Graceful Suspension | Policy violation, eval drop, maintenance, human checkpoint | 5–30s | Preserved | Minutes |
| 3 | Scope Restriction | Off-scope behavior, capability downgrade needed | <1s | Preserved | Seconds |
| 4 | Pact Suspension | Dispute filed, compliance review, audit period | Immediate | Preserved | Hours–weeks |
| 5 | Financial CB | Cost anomaly, spending freeze, financial investigation | Immediate | Preserved | Minutes–hours |
| 6 | Reputation Suspension | Sustained violations, adversarial behavior, market exclusion | Minutes | Preserved | Days–weeks |
The right architecture includes all six. The right practice is knowing which one to reach for without having to think about it. Build the decision tree into your incident runbook. Test all six mechanisms before you need any of them. Measure your RTO quarterly.
An agent economy without reliable kill-switches is not an economy—it's a liability. The agents that earn lasting trust are the ones that operators can stop cleanly and predictably, at any time, with full auditability. That predictability is not a constraint on agent autonomy. It is the prerequisite for it.