Shadow mode

What it is for

The honest objection to a context-selection service is: how do I know it would not have thrown away something my model needed? Shadow mode answers it on your own traffic instead of on ours. Needlepath runs the full computation and reports what it would have selected and what that would have saved, while you keep sending your full context to your model, unchanged. Nothing about your output moves. Nothing is at risk. You run it for a week, look at the engage rate and the savings on your real workload, and then decide.

How to run it

Shadow mode is a calling pattern, not a wire field. Call /v1/context/select as normal, record the answer, and ignore it:

selection = None
try:
    response = requests.post(
        f"https://{HOST}/v1/context/select",
        headers={"Authorization": f"Bearer {API_KEY}",
                 "content-type": "application/json"},
        json={
            "request_id": request_id,
            "records": records,
            "task": {"prompt": prompt},
            "budget": {"max_context_tokens": 4000,
                       "operating_point": "np-2026-08-r4"},
            # The rendered block comes back regardless; in shadow you
            # discard it along with the rest of the answer.
            "render": False,
            "return_per_record": True,
        },
        timeout=10,
    )
    response.raise_for_status()
    selection = response.json()
except Exception:
    pass                                    # shadow must never affect the call

if selection is not None:
    metrics.record(
        request_id=request_id,
        records_available=selection["records_available"],
        records_selected=selection["records_selected"],
        tokens_before=selection["tokens_before"],
        tokens_after=selection["tokens_after"],
        tokens_saved=selection["tokens_saved"],
        outcome=selection["outcome"],          # engaged | stood_down | escalated
        stood_down=selection["outcome"] == "stood_down",
        policy_version=selection["policy_version"],
        selected_ids=[s["record_id"] for s in selection.get("selected", [])],
    )

# Your existing call, completely unchanged.
answer = my_model.complete(prompt, context=render_everything(records))

Three rules make this safe:

Never let it affect the real call

Wrap it so that any failure is swallowed. A shadow measurement that can break production is worse than no measurement.

Ignore the rendered block

You are not going to use rendered_context. render: false is the field for saying so and it is currently ignored, so the block is built and returned regardless; discard it client-side. Send the field anyway; it starts working without a change on your side.

Run it out of band if latency matters

If your critical path cannot absorb the extra round trip, fire the shadow call asynchronously, or sample it: pick a fixed fraction by a stable hash of request_id, sized so the sample still covers every workload you care about.

What to record

Field	Why
`records_selected` / `records_available`	The selection rate, the headline.
`tokens_saved`, `tokens_before`	What you would have saved, on the engine’s own accounting.
`outcome`	How often it would have engaged versus stood down. Read this, not `gate.engaged`: `gate` is informational and can disagree with `outcome`.
`selected[].record_id`	The one that answers the real question: which of my records would have been dropped?
`policy_version`	So a week of measurements is attributable to one configuration.

The last one matters more than it looks. An aggregate savings number does not tell you whether a specific answer would have degraded: the dropped-record list does. Spot-check the turns where something you expected to survive did not.

Costs

A shadow call is a normal request: it counts against your rate limit and your daily quota, and it is priced exactly like the same request made live. What shadowing removes is the risk of changing your output, not the cost of finding out. Two consequences:

You are roughly doubling your request rate if you shadow every call. Check your limits before you start, and consider sampling.
The per-request ceiling applies as normal, so a payload that would 413 live also 413s in shadow. That is useful information: it tells you the workload needs splitting before it can be selected at all.

Get started

Release notes

How selection works

Running it in production

Billing

Integrations

What it is for

How to run it

What to record

Costs

​What it is for

​How to run it

​What to record

​Costs

What it is for

How to run it

What to record

Costs