dotnet / dotnet/runtime

HTTP/2 client sends HPACK dynamic table size update mid-header-block under concurrent load, causing backend to close connection (RFC 7541 §4.2 violation)

Open
#132,707 8 comments 0 reactions 0 assignees View on GitHub
area-System.Net.Http needs-author-action question
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Description

Under concurrent load, `SocketsHttpHandler`'s HTTP/2 client intermittently produces a malformed HPACK header block: a Dynamic Table Size Update instruction appears somewhere other than the very start of the header block. Per RFC 7541 §4.2, this instruction MUST occur only at the beginning of a header block. Apache Tomcat (acting as the backend server we proxy to) correctly rejects this as a connection error and terminates the HTTP/2 connection, cascading into `RST_STREAM`/closed-stream errors for every other in-flight request multiplexed on that connection.

This only reproduces under concurrent request load against a shared HTTP/2 connection never with concurrency = 1. This points to a race condition in how the client serializes header-block writes across concurrent streams sharing one connection's HPACK encoder state, rather than a per-request encoding bug.

### Reproduction Steps

1. Configure HttpClient/SocketsHttpHandler to talk HTTP/2 to a backend (HttpVersionPolicy.RequestVersionExact), with a shared/pooled handler so multiple concurrent requests multiplex over the same physical HTTP/2 connection.
2. Drive concurrent load against the proxy such that many SendAsync calls are in flight simultaneously over the same HTTP/2 connection to the backend concurrent threads).
3. Observe backend logs (Tomcat, org.apache.coyote.http2) for HPACK decode failures.

### Expected behavior

Each header block sent to the backend is fully HPACK-compliant: any dynamic table size update instruction appears only as the very first entry of a header block, never interleaved into an in-progress block for another stream.

### Actual behavior

Tomcat throws (server-side log, `FINE` level):

```
org.apache.coyote.http2.ConnectionException: There was an error during the HPACK decoding of HTTP headers
at org.apache.coyote.http2.Http2Parser.readHeaderPayload(Http2Parser.java:572)
at org.apache.coyote.http2.Http2Parser.readHeadersFrame(Http2Parser.java:310)
...
Caused by: org.apache.coyote.http2.HpackException: Any table size update must be sent at the start of a header block
at org.apache.coyote.http2.HpackDecoder.handleMaxMemorySizeChange(HpackDecoder.java:165)
at org.apache.coyote.http2.HpackDecoder.decode(HpackDecoder.java:153)
at org.apache.coyote.http2.Http2Parser.readHeaderPayload(Http2Parser.java:570)
... 21 more
```

This is immediately followed by cascading stream-closure errors for other in-flight requests on the same connection, e.g.:

```
org.apache.coyote.CloseNowException: Connection [1f], Stream [36185], This stream is in state [CLOSED_RST_RX] and is not writable
at org.apache.coyote.http2.Stream.doStreamCancel(Stream.java:309)
...
```

We've also intermittently seen a related but distinct HPACK framing error from the same underlying connection under load, which may share a root cause:

```
org.apache.coyote.http2.ConnectionException: Data left over after HPACK decoding - it should have been consumed
at org.apache.coyote.http2.Http2Parser.onHeadersComplete(Http2Parser.java:684)

```

### Regression?

No

### Known Workarounds

- **Setting `EnableMultipleHttp2Connections = true`: does NOT resolve the issue.** The failure still reproduces with this enabled. We believe this is expected given our root-cause theory below this setting only opens *additional* physical connections once `SETTINGS_MAX_CONCURRENT_STREAMS` is exhausted on existing connections; it does not limit the number of streams concurrently multiplexed per connection. Each individual connection still has multiple concurrent streams writing header blocks against its single HPACK encoder, so the same race is still possible on any one of the pooled connections. This result argues against a "too many streams on one connection" explanation and for a "concurrent writers against one connection's encoder, however many streams that connection is carrying" explanation.
- Falling back to HTTP/1.1 for the affected leg eliminates the issue entirely (expected, since there's no shared HPACK state in HTTP/1.1).

### Configuration

- **.NET version:** 10.0.11
- **OS:** Windows Server 2022 Standard
- **HttpClient configuration:**
```csharp
var httpRequest = new HttpRequestMessage(method, uri)
{
Version = HttpVersion.Version20,
VersionPolicy = HttpVersionPolicy.RequestVersionExact
};
```
Client obtained via `IHttpClientFactory` / pooled `HttpMessageHandler` (handler reused across concurrent requests, default `EnableMultipleHttp2Connections` = false at time of repro).
- **Backend server:** Apache Tomcat 10.1.46, HTTP/2 (h2), NIO connector
- **Load pattern:** Reproduced under JMeter load test with 60 concurrent threads issuing requests through a .NET reverse proxy sitting in front of Tomcat. Not reproducible at concurrency = 1.

### Other information

- We ruled out header **content** as the cause: stripping `User-Agent` and other client-controlled headers from the forwarded request does not prevent the failure. The failure persists with a minimal, stable set of forwarded headers.
- The failure is highly load-dependent: never seen at low concurrency, reproducible reliably at 60 concurrent threads against a single pooled `HttpClient`/HTTP/2 connection.
- We noted in `Http2Connection.cs` a comment indicating the client does not currently support dynamic table indexing when *sending* requests ("This approach must be revisited if we ever support the dynamic table or compression when sending requests"), which suggests the client's outbound HPACK path is not fully hardened for concurrent multiplexed writes — we suspect the bug lies in how/when a table-size-update instruction (sent e.g. in response to a `SETTINGS_HEADER_TABLE_SIZE` change or connection-level state change) gets attached to an outgoing header block when multiple streams are concurrently writing headers to the same connection.
- We confirmed this is not simply a function of too many streams sharing one connection: enabling `EnableMultipleHttp2Connections = true` does not resolve it (see Workarounds below). Since that setting only adds new connections once `MAX_CONCURRENT_STREAMS` is exhausted rather than limiting concurrent streams per connection, this is consistent with the race occurring per-connection regardless of how many connections exist, whenever more than one stream on a given connection is concurrently writing headers.

Contributor guide

Open the contributing guide

Research direction

Begin with Http2Connection.cs, especially the noted outbound HPACK and dynamic-table handling, and trace how concurrent streams serialize header blocks on one connection. Reproduce with the shared pooled HttpClient, HTTP/2 exact version, and concurrent load against Tomcat, then verify that HPACK decoding no longer reports mid-block table updates or related framing errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.