From nine documents lost to zero

This is the engineering write-up that sits where a competitor would put a benchmark table. It is about throughput and loss rather than answer quality, it is measured on our own corpus, and every number in it is a before-and-after on the same system rather than a comparison with anyone else.

The four numbers

The last row is the one to read first, because it explains the first. A step consuming three hundred and forty-eight percent of the timeout budget means the request was killed before the work finished — every single time — and the work was then abandoned rather than resumed.

The failure that does not look like a failure

Nine documents out of two hundred and five were lost. Not failed — lost. They were accepted with a success response, they entered a job, the job died partway through, and nothing retried.

From the caller's side: a 200, and then no document. No error to surface, no failed row to investigate, nothing in a dead-letter queue because there was no queue. The only way to detect it was to count what you sent and count what arrived, which nobody does until they have a reason to.

What was actually wrong

Four things, and only one of them was slow code.

Retries were at the wrong granularity

A document was one job. Extraction within it was sharded — the text split into pieces, each piece sent to the model. When one shard failed, the retry re-ran the entire document, including the shards that had already succeeded. That is wasteful, and worse, it meant a document with a persistently awkward shard could never complete: every attempt re-did the expensive successful work before hitting the same wall.

Long work ran inside a request

Extraction happened in the process handling the HTTP call. Any load balancer has a timeout, and a hundred-second p90 with a much longer tail comfortably exceeds a typical one. The connection was severed and the in-flight work went with it.

There was no back-pressure

A model provider rate limit produced a failure rather than a wait. There was no mechanism to pause and resume — the only options in the code were succeed now or fail — so a burst of ingest during a rate-limited window simply lost documents.

Concurrency was unbounded per tenant

One organisation backfilling could occupy roughly seventy percent of platform extraction capacity, and nothing prevented it. Everyone else's ingest slowed to a crawl, which from their side looked like the service being slow rather than a neighbour being loud.

What fixed it

Moving orchestration onto durable steps. The specific properties that mattered, in order of how much they contributed:

1. Step-level retries. Each shard is its own step with its own retry policy, so a failed shard retries alone and the successful ones are not re-run. This is what eliminated the persistently-stuck document.
2. Work outside the request. Ingest accepts, enqueues, and returns a status URL. Nothing long-running lives inside an HTTP request any more, so the load-balancer timeout stopped being a factor at all — which is what took the worst step from 348% of the budget to 4%.
3. Durable waits. A rate limit becomes a pause rather than a failure. The run sleeps and resumes; the document arrives late instead of not arriving.
4. Per-organisation concurrency ceilings, plus a token budget metered in the unit the model provider actually bills. This is what stopped the noisy-neighbour problem, and metering in the provider's own unit rather than an approximation of it is what made the ceiling meaningful.
5. Replayable runs. A bad deploy is recoverable by replaying the affected runs rather than by asking callers to re-send.

The throughput improvement — three to seventeen documents a minute — came mostly from the second and fourth items rather than from anything getting faster. Work that is not being abandoned and retried from scratch completes a great deal sooner.

The p90 number, honestly

Extraction p90 going from about a hundred and three seconds to 5.2 seconds looks like a twenty-fold speed-up and it is not one. The model calls did not get twenty times faster.

Most of that number is the removal of retry amplification. When a document's p90 includes the time spent re-running work that had already succeeded, the measurement is dominated by waste rather than by work. Sharding properly with independent retries removes the waste, and what is left is closer to the true cost of extracting a document.

It is a real improvement in wall-clock time for a caller. It is not evidence that the extraction step itself was optimised, and presenting it as such would be the kind of thing this page exists to avoid.

What this does and does not tell you

It tells you that the pipeline is no longer the source of missing documents, that a backfill will complete, and that one tenant cannot starve another. Those are the properties that determine whether a retrieval system is trustworthy in a way that has nothing to do with relevance.

It tells you nothing about answer quality. Extraction completing reliably is a prerequisite for good answers and not a cause of them. For what we have and have not measured on quality, the methodology page is the honest account, and it is less flattering than this one.

If you are building this yourself

The four fixes above are the work, and none of them is novel. Budget for them at the start rather than discovering them the way we did:

• Shard extraction and retry per shard, not per document.
• Never run extraction inside an HTTP request. Accept, enqueue, return a status URL.
• Make rate limits a wait rather than a failure.
• Bound concurrency per tenant before you have a second tenant, not after.
• And count what you accepted against what arrived, continuously. That reconciliation is the only thing that detects silent loss, and it is the check we did not have.