Skip to main content

​
Two error shapes, and how to tell them apart

Some failures are produced by the service; some are produced by the gateway in front of it, before the service is reached. They do not share a body shape.
Match on the HTTP status code, and — for service errors only — on error.
  • error is a stable machine-readable code: bad_request, request_too_large, engine_error, not_found.
  • detail is for a human reading a log. It is not stable; do not parse it.
  • The gateway’s key is capitalised inconsistently between cases (message in some, Message in others) because those are the platform’s own bodies passed through unchanged. Do not build on that key at all.

​
The table

​
401 vs 403

These mean genuinely different things and are worth wiring to different alerts.

401 — the header was missing entirely

No Authorization header at all. Rejected at the gateway before anything of ours runs. This is almost always a deployment mistake: an unset environment variable, a proxy stripping the header, a misspelled header name.

403 — a header arrived and was refused

Anything else. Including an Authorization header that is present but not Bearer-shaped — that reaches authorization and is denied, it does not become a 401. Other causes: an unknown, revoked or malformed key; a suspended organisation; a control plane too stale to vouch for the key.
The 403 body deliberately does not say which cause applied. An error that distinguishes “no such key” from “suspended organisation” is an oracle for anyone probing keys — so the message is uniform on purpose, and the way to find out which one you hit is to ask us.
A 403 right after minting a key is usually a provisioning race, not a bad key. Associating a new key with its plan is not instantaneous — it can take a minute or two. Retry once after a short delay before concluding anything.

​
Retrying a 429

Both a rate limit and a daily quota surface as 429, and they want opposite behaviour: Branch on the body here if you can — it is the one place a gateway message is worth reading, because the status code alone cannot distinguish “retry shortly” from “retry tomorrow”. Treat it as a hint rather than a contract: it is the platform’s string, not ours, so default to capped exponential backoff when the body is missing or unrecognised, and let a circuit breaker stop the retries rather than relying on the message to tell you to. Your plan limits are approximate in both directions (why), so a 429 is a normal operating condition, not an incident. Build backoff in from the first day.

​
402: not serving yet

402 is reserved for “your organisation is out of credit”, and is documented here ahead of time so client libraries reserve the case rather than folding it into 403. Until it ships, a credit problem does not produce a distinct status.
Reserve the case; do not test against it. Nothing returns 402 today.

​
The two cases that are not errors

​
Reading a stand-down

Needlepath can decide that trimming would not pay and hand your context back essentially intact. This is a 200 with:
Send what you were going to send. A stand-down is the engine declining to make things worse, and on some workload shapes it is the common outcome rather than a rare one — do not alarm on it and do not treat a tokens_saved of zero as a failure.
fallback_used: true does not tell you why. Three different things produce it:
  1. a deliberate gate stand-down — gate.engaged is false with a reason;
  2. an empty or unusable record set;
  3. an engine-internal exception — selection_error is populated.
If you care about the difference, read gate, selection_error and format_metrics.engine_fallback_reason. fallback_used alone conflates a healthy decision with an outage.
Two more traps in the same area:
  • safety is null on the plain fixed-budget path, because no coverage verdict is computed there. A null safety means “no verdict ran”, never “the verdict was fine”.
  • An empty selected[] is not proof that nothing was selected. It is also empty when you sent return_per_record: false. Read records_selected.

​
An engine failure returns 200, not 5xx

If the selection pass itself raises, you get a 200 with selection_error populated and a usable full-context body — because the correct client behaviour in that case is to send your original context, and the body already contains it.
This decides how you monitor us. An availability metric defined on “non-5xx” will report perfect health straight through an engine outage. Alarm on selection_error being non-null, not on status code.
A 500 is reserved for the cases where no response could be produced at all. Two shapes, and they want different handling:
  • {"error": "engine_error", …} — ours. If the cause is one of the two rough edges below (an unresolvable operating point, an unknown kind) it is deterministic and retrying it unchanged will fail identically. Fix the request instead.
  • No error key — the platform’s, produced when something in front of the service fails. Retry that one with capped exponential backoff.
Since you cannot always tell in advance which you have: retry a 500 at most a couple of times with backoff, and stop. A 500 that survives two attempts on an unchanged request is not going to clear on the third.

​
Two rough edges worth knowing

Both currently surface as a 500 where a 400 would be more informative. They are behaviour, not intent — documented so you can recognise them in a log rather than debug them twice. Neither is retryable: the same request fails identically. Read /v1/operating-points for valid labels, and the record kinds for valid kinds.

​
Timeouts and fail-open

Set a client timeout you are happy to add to your critical path. Every response reports the server-measured selection time as engine_latency_ms; your round trip adds network transit on top of it, and the two are different measurements that should not be added together casually. Measure both from your own client before you choose a timeout — a 10-second default is generous for either. Whatever the failure — timeout, non-2xx, unparseable body, empty selection — send your original context unmodified. Selection is an optimisation, and a failed optimisation degrades to “no optimisation”, never to an empty prompt. There is a worked implementation in the quickstart.