Shadow mode

What it is for

The honest objection to a context-selection service is: how do I know it would not have thrown away something my model needed? Shadow mode answers it on your own traffic instead of on ours. Needlepath runs the full computation and reports what it would have selected and what that would have saved — while you keep sending your full context to your model, unchanged. Nothing about your output moves. Nothing is at risk. You run it for a week, look at the engage rate and the savings on your real workload, and then decide.

How to run it

Shadow mode is a calling pattern, not a wire field. Call /v1/context/select as normal, record the answer, and ignore it:

selection = None
try:
    response = requests.post(
        f"https://{HOST}/v1/context/select",
        headers={"Authorization": f"Bearer {API_KEY}",
                 "content-type": "application/json"},
        json={
            "request_id": request_id,
            "records": records,
            "task": {"prompt": prompt},
            "budget": {"max_context_tokens": 4000,
                       "operating_point": "np-2026-07-r2"},
            # The rendered block comes back regardless; in shadow you
            # discard it along with the rest of the answer.
            "render": False,
            "return_per_record": True,
        },
        timeout=10,
    )
    response.raise_for_status()
    selection = response.json()
except Exception:
    pass                                    # shadow must never affect the call

if selection is not None:
    metrics.record(
        request_id=request_id,
        records_available=selection["records_available"],
        records_selected=selection["records_selected"],
        tokens_before=selection["tokens_before"],
        tokens_after=selection["tokens_after"],
        tokens_saved=selection["tokens_saved"],
        engaged=(selection.get("gate") or {}).get("engaged"),
        stood_down=selection["fallback_used"],
        policy_version=selection["policy_version"],
        selected_ids=[s["record_id"] for s in selection.get("selected", [])],
    )

# Your existing call, completely unchanged.
answer = my_model.complete(prompt, context=render_everything(records))

Three rules make this safe:

Never let it affect the real call

Wrap it so that any failure is swallowed. A shadow measurement that can break production is worse than no measurement.

Ignore the rendered block

You are not going to use rendered_context. render: false is the field for saying so and it is currently ignored, so the block is built and returned regardless — discard it client-side. Send the field anyway; it starts working without a change on your side.

Run it out of band if latency matters

If your critical path cannot absorb the extra round trip, fire the shadow call asynchronously, or sample it — one request in ten is enough to characterise a workload.

What to record

Field	Why
`records_selected` / `records_available`	The selection rate — the headline.
`tokens_saved`, `tokens_before`	What you would have saved, on the engine’s own accounting.
`gate.engaged`, `fallback_used`	How often it would have engaged versus stood down.
`selected[].record_id`	The one that answers the real question: which of my records would have been dropped?
`policy_version`	So a week of measurements is attributable to one configuration.

The last one matters more than it looks. An aggregate savings number does not tell you whether a specific answer would have degraded — the dropped-record list does. Spot-check the turns where something you expected to survive did not.

What shadow mode will add

When the wire field ships, the differences from doing it by hand are:

One switch, not a second call path. A flag on the request rather than a parallel branch in your code.
A report you do not have to build. Per-workload engage rate, projected savings and projected cost, without you aggregating anything.
Priced identically to a real selection, deliberately — the same request in shadow costs what it would have cost live, which is what makes the projection comparable to the bill it predicts rather than an optimistic estimate.

That last point is worth reading twice: shadow mode will not be free. It does the full computation, so it costs the full computation. What it removes is the risk of changing your output, not the cost of finding out.

Costs, doing it by hand

A manual shadow is a normal request, so it counts against your rate limit and your daily quota exactly like any other. Two consequences:

You are roughly doubling your request rate if you shadow every call. Check your limits before you start, and consider sampling.
The per-request ceiling applies as normal, so a payload that would 413 live also 413s in shadow. That is useful information — it tells you the workload needs splitting before it can be selected at all.

Get started

How selection works

Running it in production

Billing

Integrations

What it is for

How to run it

What to record

What shadow mode will add

Costs, doing it by hand

​What it is for

​How to run it

​What to record

​What shadow mode will add

​Costs, doing it by hand

What it is for

How to run it

What to record

What shadow mode will add

Costs, doing it by hand