Detection, containment and recovery catalogue
Last verified · 2026-07-1612 · Failure Modes
A failure is specified only when the system can detect it, contain its authority, recover or escalate, explain the user-visible state and emit actionable telemetry.
12.01 · Policy
Unknown financial state stops new risk
Provider failure produces no new intent. Unknown signing/exchange state pauses risk and reconciles before retry. Match state reloads from persisted revision/timestamps. Settlement decisions are immutable and effects retry idempotently. Replay corruption never changes authority.
12.02 · Catalogue
Required operational response
| Failure | Detection | Containment | Recovery | User state | Telemetry |
|---|---|---|---|---|---|
| Model timeout | Turn deadline exceeded | No intent; keep prior exposure | Retry next cadence only if state remains fresh | Agent delayed; controls remain active | turn_timeout, provider, session, duration |
| Invalid model output | Schema/semantic validation fails | Reject before risk/signing | Return bounded correction once, then wait | Decision rejected as invalid | intent_invalid, safe validation codes |
| Duplicate tool invocation | Repeated intent/request idempotency key | Return prior result; do not resubmit | Reconcile prior request state | No duplicate action | duplicate_intent, original request |
| Signing failure | Signer/policy service error | No exchange submission; pause writes after threshold | Verify signer registration/expiry, rotate through master path | Trading temporarily unavailable | sign_failure without key material |
| Stale market data | Freshness exceeds policy | Reject risk-increasing intents | Reconnect and fetch authoritative snapshot | Market data stale; reduce/exit controls prioritized | market_stale, feed, age |
| WebSocket disconnect | Close/heartbeat timeout | Mark projection stale | Reconnect, consume snapshot, reconcile info endpoints | Reconnecting; state may be delayed | ws_disconnect, attempts, gap |
| Missed fill | Reconcile result differs from projection | Freeze new risk if exposure uncertain | Insert deduplicated fill and rebuild portfolio | Portfolio reconciling | fill_gap, trade/order IDs |
| Nonce conflict | Exchange nonce rejection/duplicate | Stop signer submissions | Single-writer atomic counter fast-forward; rotate only by runbook | Execution paused for signer recovery | nonce_conflict, signer reference |
| Partial order execution | Remaining size/open status after fill | Recompute exposure from fills | Policy decides cancel/reprice/reduce; never assume full fill | Partially filled | partial_fill, filled/remaining |
| Cancel failure | Cancel rejected/unknown and order remains open | No conflicting new risk | Query order status; retry idempotently until deadline/escalation | Cancel pending verification | cancel_failure, order ID |
| Flatten failure | Open position/order after flatten plan | Terminal state becomes flatten_failed/manual_review | Repeat cancel and reduce-only plans with fresh sizes; operator escalation | Exit incomplete; recovery active | flatten_failure, residual exposure |
| Hyperliquid outage | HTTP/WS error budget exceeded | Stop new risk; preserve authoritative local state | Backoff, health probe, reconcile before resume | Exchange unavailable | venue_outage, endpoint/status |
| Privy outage | Auth/wallet API failures | Existing verified sessions may read; sensitive wallet actions fail closed | Backoff and reauthenticate; never bypass policy | Wallet service unavailable | identity_outage, operation |
| Cloudflare restart | New isolate/object activation | Never rely on memory for critical state | Load persisted revision, reconcile pending effects | Usually transparent; recovery state if needed | activation, recovered revision |
| Durable Object migration | Schema version mismatch/migration error | Reject state mutations | Backward-compatible migrator, backup/export, canary and rollback | Match temporarily unavailable | do_migration, from/to/version |
| Settlement retry | Workflow/payout result unknown or failed | Decision immutable; no new decision | Retry effect with decision idempotency key; reconcile ledger | Settlement pending | settlement_retry, decision/effect IDs |
| Replay corruption | Manifest checksum/gap fails | Replay labeled incomplete; authority unaffected | Rebuild from indexed source events or retain evidence gap | Replay temporarily incomplete | replay_corrupt, segment/hash |
| User disconnection | Client WebSocket closes | Server session/match continues by policy | Reconnect with latest snapshot/revision | Rejoining live state | client_disconnect, role/duration |
| Opponent disconnection | Opponent connection absent | Do not change timer or trading authority automatically | Apply published abandonment rule; reconnect snapshot | Opponent offline; match continues/terminal rule shown | opponent_disconnect, match/revision |
| Reference-price unavailable | Required price sources stale/missing at snapshot | No winner decision | Use preapproved fallback hierarchy or manual_review/refund | Settlement review required | reference_price_missing, sources |
12.03 · Verification
Fault drills are release evidence
Inject provider timeouts, malformed tool calls, stream gaps, unknown submissions, partial fills, cancel/flatten failures, exchange/Privy outage, isolate/DO restart, alarm delay, queue duplicate/DLQ, settlement retry and replay corruption. Record state transitions, exchange/account reconciliation, alerts, user UI and runbook results.