microsoft / microsoft/fhir-server

Bundle batch/transaction: response stream reused across throttle-retry re-invoke, corrupting body-bearing 429 responses

Open
#5,670 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Up for grabs
Dominant language
TSQL
Stars
1.4k
Forks
592
Avg merge
2d 7h
Merged PRs (30d)
41

Description

Summary

BundleHandler's throttling/retry logic re-invokes the inner request handler on the same httpContext.Response.Body MemoryStream for the retry attempt after a 429, without resetting it first. If the handler writes a body on both the throttled attempt and the retry attempt (which is exactly what TooManyRequestsActionResult does in real production — it returns an OperationOutcome body), the second write appends after the first, producing malformed, concatenated JSON in the recorded response body for that bundle entry.

This is not specific to any one FHIR SDK path — it's in the shared bundle-processing engine, unrelated to any content-negotiation or serialization concerns.

Where

Two call sites, both affected identically:

  • BundleHandler.ExecuteRequestsWithSingleHttpVerbInSequenceAsync (sequential execution path)
  • BundleHandlerParallelOperations.ExecuteRequestsInParallelAsync (parallel execution path)

Both re-invoke the inner request handler for the retry attempt against the same httpContext.Response.Body stream used for the initial (throttled) attempt, with no Seek/reset/reallocation in between.

Reproduction / discovery context

Found while writing test coverage for a bundle entry that receives a 429 (with a real OperationOutcome body, matching what TooManyRequestsActionResult actually produces) and is subsequently retried. Writing a body on both the throttled attempt and the retry attempt causes the second write to append onto the stream instead of replacing it, so downstream parsing of the recorded body fails — the exact symptom depends on which JSON parser reads it back (e.g. a System.Text.Json JsonException: '{' is invalid after a single JSON value, or a FormatException from FhirJsonParser), but the root cause is the same stream-reuse bug in both cases.

Suggested fix

Reset the response stream (and status code) before each retry re-invoke at both call sites — something like:

httpContext.Response.Body = new MemoryStream();
httpContext.Response.StatusCode = 200;

immediately before the retry invocation, at both the sequential and parallel call sites, with dedicated test coverage for a body-bearing 429 followed by a body-bearing retry response.

Scope note

This predates any recent SDK-migration work in this codebase — the retry-reinvoke code has no SDK-mode branching around it and affects every code path that goes through this shared engine equally. It's a genuine reliability bug in its own right, not a parity gap between implementations, so it deserves its own dedicated fix-and-review cycle rather than being folded into unrelated work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with BundleHandler.ExecuteRequestsWithSingleHttpVerbInSequenceAsync and BundleHandlerParallelOperations.ExecuteRequestsInParallelAsync, focusing on the retry invocation and the reused httpContext.Response.Body stream. Add coverage for a body-bearing 429 followed by a body-bearing retry response in both sequential and parallel paths. Done means the recorded response body contains only the retry response and downstream parsing succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.