Error Classification & Retry Cascade

ClawRouter classifies upstream errors to determine the correct recovery action. Each error type triggers a specific behavior in the retry cascade.

Version 1.0.18


Error Classification Reference

Error TypeHTTP StatusBody PatternsRecovery Action
AUTH_ERROR401, 402, 403 (no model/subscription patterns)API_KEY_INVALID, FAILED_PRECONDITION, hard-billing bodies ("credit balance", "insufficient balance"), 429 with insufficient_quota, Z.AI codes 1000-1005 / 1113 / 1309 / 1314 / 1315Disable key permanently, try next key
QUOTA_EXHAUSTED429, 403, 400, 401Window-quota bodies: "usage limit", "access_terminated", "billing cycle", "quota will be refreshed", "quota exceeded", "usage_limit_reached"; Z.AI codes 1308 / 1310 / 1316-1321Window-accurate backoff on key (until its own window's reset), never disabled, try next key
RATE_LIMIT429 (transient bodies), 498 (Groq)Rate limit patterns60s backoff on key, try next key
MODEL_ERROR404, 400, 401, 403"model", "ModelError", "PAID_MODEL_AUTH_REQUIRED", "requires a subscription"Try next model (if enabled), then next key
OVERLOADED503, 529 (Anthropic)"overloaded", "capacity", "resource exhausted"Wait 2s, retry same key (up to 2x)
REQUEST_ERROR400, 413, 422, 499Bad format, too large, unprocessableReturn to client immediately
SERVER_ERROR500, 502, 503 (non-overload)Server-side transientTry next key
TIMEOUT504, 408, timeout/AbortErrorNetwork timeoutTry next key (if retry_on_timeout)
NETWORK_ERROROtherConnection failuresWait 1s, retry same key (up to 3x), then exhaust to fallback

Retry Cascade (Exact Order)

When a request fails, ClawRouter follows this exact retry cascade:

  1. REQUEST_ERROR (400/413/422/499) > Return to client immediately (will fail with any key).
  2. OVERLOADED (503/529) > Wait 2 seconds, retry same key (up to 2 inner retries).
  3. MODEL_ERROR (404/400/401/403 with model or subscription patterns) > Try next model from provider_models if model_fallback_enabled (same key). Repeated model failures trip the model circuit breaker, after which the model is skipped entirely -- see Model Fallback.
  4. AUTH_ERROR / QUOTA_EXHAUSTED / RATE_LIMIT / SERVER_ERROR / TIMEOUT / NETWORK_ERROR > Try next key.
  5. All keys exhausted > Trigger Provider Fallback Chain (tried in priority order).
  6. All fallback providers exhausted > Return error to client.

Key Backoff & Error Handling Summary

Error TypeKey ActionBackoff DurationNotification
Rate Limit (429, Groq 498)Temporary cooldown60 seconds (configurable)Key Rate Limited
Quota Exhausted (window quota: Kimi Coding 5h/weekly/monthly cycles)Long cooldown -- never disabledUntil the key's own exhausted window resets (see below)Key Rate Limited
Auth Error (401, 402, hard-billing bodies)Permanently disabledPermanentKey Disabled
Overloaded (503, 529)Retry same key2 seconds (up to 2 retries)None
Model ErrorNo key actionN/AModel Fallback (if enabled)
Request Error (400, 413, 422)No key actionN/ANone
Server Error (500, 502)Try next keyN/ANone
Timeout (504, 408)Try next key (if retry_on_timeout)N/ANone
Network ErrorRetry same key (up to 3x)1 second between retriesNone

Window Quota vs Hard Billing

Two kinds of "out of quota" errors exist, and ClawRouter treats them very differently:

  • Window quota (QUOTA_EXHAUSTED) -- self-resetting cycles: Kimi Coding's 5-hour/weekly/monthly windows, Z.AI's 5-hour/7-day windows (codes 1308/1310/1316-1321), OpenAI monthly quota. The key is valid; its quota window is simply full. The key enters a long backoff and is never disabled. It recovers automatically when the window resets.
  • Hard billing (AUTH_ERROR) -- the account has no money left: HTTP 402, "credit balance too low", MiniMax insufficient_balance, Z.AI code 1113 / insufficient_quota (a 429!). These disable the key permanently, like any auth error.

The distinction matters because window-quota errors often arrive on auth-looking statuses -- Kimi Coding returns HTTP 403 access_terminated_error, not 429 -- and naively treating every 403 as an auth error would disable a perfectly good key.

How the Quota Cooldown Is Calculated

Cooldowns are per-key and window-accurate -- not a shared blanket cooldown. When a key hits QUOTA_EXHAUSTED, ClawRouter picks the reset time in this order:

  1. The key's saved quota snapshot -- every successful quota probe (Quota tab or traffic-time) persists that key's windows and reset times. The error text is matched to a window kind (5-hour vs weekly), so a key that exhausts its 5-hour window cools down until its 5-hour reset, and a weekly exhaustion cools down until its weekly reset. Two keys tripping different windows at the same moment get different, accurate cooldowns.
  2. A reset time parsed from the error body (capped at quota_backoff_s).
  3. Monthly billing-cycle detection -- a "usage limit ... next cycle" style body means the monthly cycle is exhausted even when 5-hour/weekly quota shows available. The message is "Monthly usage cycle exhausted"; the key is not disabled -- it is retried in ~10-day steps anchored to its last successful use until the cycle renews.
  4. A live usage-endpoint probe for the actual reset time.
  5. Default quota_backoff_s (1800 seconds, configurable in Settings) when nothing else is known.

Trusted reset times (snapshot, probe, monthly anchor) bypass the quota_backoff_s cap -- a weekly window really is days long -- with a 30-day sanity ceiling.


Provider-Specific Quirks

Some providers return non-standard HTTP status codes that ClawRouter handles specially:

ProviderQuirkHow ClawRouter Handles It
OpenCode.aiReturns HTTP 401 for unsupported models (body: "ModelError")Classified as MODEL_ERROR, not AUTH_ERROR. Key is not disabled.
Kilo.aiReturns HTTP 401 for non-free models when no auth (body: "PAID_MODEL_AUTH_REQUIRED")Classified as MODEL_ERROR.
Ollama CloudReturns HTTP 403 "this model requires a subscription, upgrade for access" for plan-gated modelsClassified as MODEL_ERROR. The free-tier key stays enabled; model fallback and the model circuit apply.
Kimi CodingReturns HTTP 403 access_terminated_error (not 429) when a 5-hour, weekly, or monthly quota window is fullClassified as QUOTA_EXHAUSTED. Key backed off until its own exhausted window's reset (per-key snapshot), never disabled.
Z.AIPuts a numeric-string business code in error.code that beats the HTTP status -- hard billing, window quotas, model gating, and rate limits all ride on 429Code-driven classification (table below). A 429 can disable the key (1113), back it off (1308-1321), or just rate-limit it (1302/1313).
Google GeminiReturns HTTP 400 for invalid API keys (body: "API_KEY_INVALID")Classified as AUTH_ERROR. Key is disabled.
MiniMaxReturns HTTP 200 for most errors with custom status codes in bodyParsed from response body (1004/2049/1008 = AUTH_ERROR, 1002/2045/2056 = RATE_LIMIT).
AnthropicUses custom HTTP 529 for overloadedClassified as OVERLOADED.
GroqUses custom HTTP 498 for flex tier capacityClassified as RATE_LIMIT.

Z.AI Business Codes

Z.AI returns a numeric-string code in error.code that overrides the HTTP status (almost everything arrives as 429). ClawRouter classifies by code, per Z.AI's published error table:

CodeMeaningClassificationKey Action
1113Insufficient balance (hard billing)AUTH_ERRORDisabled -- recharge required
1308, 1310, 1316-1321Self-resetting 5-hour / 7-day quota windowsQUOTA_EXHAUSTEDLong backoff, never disabled
1309, 1314, 1315Expired / wrong planAUTH_ERRORDisabled
1311Plan doesn't include the modelMODEL_ERRORModel fallback
1302, 1313Rate limitRATE_LIMIT60s backoff
1305OverloadedOVERLOADEDRetry same key
1211Unknown modelMODEL_ERRORModel fallback

Dead-key trap: Z.AI's monitor endpoint returns HTTP 200 with {"code":1000,"msg":"Authentication Failed","success":false} for a dead key. ClawRouter detects this envelope (auth-family codes 1000-1005, with or without the message) and treats it exactly like a real 401.