Debug a Gateway 400, 402 or 429

---name: lobstack-gateway-errorsdescription: "Diagnose a Lobstack Gateway failure from its status and error class and decide whether retrying can possibly help. Use when a call returns 400, 401, 402, 403, 429 or 5xx, when a stream ends in an error frame, or when someone reports that the Gateway is broken."license: MITmetadata:  title: "Debug a Gateway 400, 402 or 429"  version: 1.0.0  author: lobstack  category: gateway  tags: [gateway, errors, retries, debugging]--- # Debug a Gateway 400, 402 or 429 Every failure has the same shape, and the type field is the whole diagnosis: ```json{ "error": { "message": "…", "type": "quota", "code": 402,             "request_id": "8f2b1e0c-4d5a-4b91-9c3e-6a7f0d2b5c41" } }``` request_id is also returned as x-lobstack-request-id, on every responseincluding ones that fail before authentication finishes. It is the id of the tracerow, which holds the status, the latency, the model, the credential and the first1,000 characters of the provider's message. Quote it and a description becomes alookup; lose it and you are guessing. ## The six classes | type | status | whose problem | retry || --- | --- | --- | --- || `auth` | 401, 403 | the credential | never || `quota` | 402, 429 | allowance or rate limit | 402 no, 429 yes || `validation` | 400, 409, 422 | the request body | never || `provider` | 502, 503, other 5xx | the upstream provider | yes, with backoff || `timeout` | 408, 504 | we gave up waiting | yes, carefully || `internal` | 500 | ours | once, then report the id | An unrecognised throw is classified internal, not provider — blamingupstream for our own crashes would make the error rate look better than it is. Sointernal really does mean a Lobstack bug. ## auth — 401, 403 Causes, in the order they actually occur: no bearer token; a key that does notmatch lsk_<env>_<8 hex><48 hex>; a key that is unknown, revoked or expired; akey without the inference scope; an agent credential whose machine is notrunning, which returns 403 with the machine status in the message. Revoked and expired are reported distinctly from unknown, because "this key wasrevoked" beats "unauthorized" at two in the morning. All of them are rejections:never retry. The 401 that is not about your key. If the message reads like missingcredentials and the key is definitely good, check the host. The bare apex 307s tothe www host, and RFC 9110 requires every HTTP client to drop theAuthorization header when a redirect changes host. Usehttps://www.lobstack.ai/api/gateway/v1. ## quota — 402 and 429 are opposite advice 402 — the allowance for the period is gone. retry-after is present and iscomputed from the period reset, so it is usually days. It means "this will notclear on its own before then", not "back off". Top up, upgrade, or wait for thereset. Read x-lobstack-quota-meter first: spend carries dollar headers,requests and legacy carry integer ones, and the absent set is absentdeliberately so a client cannot read a missing header as zero allowance. 429 — a rate limit, in practice the provider's, passed through with itsstatus. Back off exponentially with jitter and retry. Enforcement is asymmetric on purpose: API keys are enforced, agent credentials getthe headers and are not blocked. So a 402 on an agent credential is not somethingyou will see today, and your own budget check is what stops the loop. ## validation — 400 It will fail identically until the body changes. Causes: messages missing orempty; a body that is not JSON; a model key the registry does not know, includingafter alias resolution; a model that exists but is not available in managed mode;BYOK mode with no key on the agent. Provider 400s also land here, with the provider's own message inside — whichmeans a validation error can be about something you never wrote. ### The worked example: the temperature 400 Anthropic deprecated the sampling parameters on Opus 4.7 and everything after it,including Opus 5, Sonnet 5 and the Fable line. The API does not ignore atemperature on those models. It returns: ```json400 {"type":"invalid_request_error","message":"`temperature` is deprecated for this model."}``` This was the most common failure in this Gateway's history, and not becausecallers were setting temperatures: the Gateway itself sent temperature: 0.7 onevery Anthropic request whether or not anyone asked for one, and every single onefailed — for months, while almost no inference ran and nobody could see why. Both halves are fixed. A temperature is forwarded only when the caller set one,and only to models that still accept it, matched on the provider-native id thatgoes on the wire. When one is dropped you getx-lobstack-dropped-params: temperature and a successful response. So if you see this 400 today, you sent a temperature to a model that rejects itthrough a path that is not this Gateway. Find that path — seeaudit-unmetered-model-calls. ## provider — 502, 503, 5xx The upstream failed and it is not your fault. The network call threw (502), theprovider returned a 5xx (passed through with its status), or no managed providerkey is configured on the deployment (503). A provider 401 is deliberately remapped to 502: Lobstack's own credential beingwrong is our failure, and reporting it as 401 would send you to check a key thatis fine. Retry with exponential backoff and jitter — except the 503 whose message says nomanaged provider key is set. That one is configuration and will fail forever untila key is added or the caller switches to BYOK. ## timeout — 408, 504 Provider calls abort at 120 seconds by default; the route allows up to 300.Anything whose message looks like an abort or a timeout lands here even when itarrives as an unrecognised throw. There is no idempotency key. A request that timed out may have completed at theprovider, so an automatic retry can produce a second answer and a second charge —both metered, because the tokens were really generated. For anything expensive,read the trace by request id before retrying rather than retrying blind. ## Failures inside a stream A stream that fails part-way through cannot change its status: it is already 200and the headers are long gone. The Gateway sends an error object into the streamand then [DONE]: ```data: {"error":{"message":"provider error 529: overloaded","type":"gateway_error","code":529,       "request_id":"8f2b1e0c-…"}} data: [DONE]``` Note that inside the stream type is gateway_error, not one of the six classnames. The class is still on the trace row. **A client that only checks the HTTPstatus will record this stream as a success** — check for an error key on everyframe. ## Counting them GET /api/v1/usage returns summary.errors_by_class over any range, which isthe fastest way to answer "was that bad hour upstream or ours". A failed round alsowrites a ledger row with zero tokens, so error rate is computable from the ledgeralone. See lobstack-usage-reporting. ## Triage in one pass 1. Pull request_id out of the response or the headers. Log it.2. Read type. It tells you whose problem it is before you read the message.3. auth or validation → fix the call. Retrying is guaranteed to fail.4. quota → check x-lobstack-quota-meter; 402 top up, 429 back off.5. provider or timeout → back off and retry, and remember there is no   idempotency key.6. internal → retry once, then report the request id.