maxConcurrentStreams is not enforced by servers

Expected behavior

When an Http2MultiplexCodec is created with maxConcurrentStreams set to N, any subsequent calls to http2MultiplexCodec.connection().numActiveStreams() should report a value no greater than N; clients should receive RST_STREAM errors for any streams they attempt to create beyond that limit.

Actual behavior

In my case (where N=2), I recently observed the following (using the -m 5 flag to nghttp, which repeats the request five times using as much parallelism as possible):

[  0.012] Connected
The negotiated protocol: h2
[  0.210] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[  0.210] send HEADERS frame <length=112, flags=0x04, stream_id=1>
          [omitted]
[  0.210] send HEADERS frame <length=14, flags=0x04, stream_id=3>
          [omitted]
[  0.210] send HEADERS frame <length=14, flags=0x04, stream_id=5>
          [omitted]
[  0.210] send HEADERS frame <length=14, flags=0x04, stream_id=7>
          [omitted]
[  0.210] send HEADERS frame <length=14, flags=0x04, stream_id=9>
          [omitted]
[  0.210] send DATA frame <length=14, flags=0x01, stream_id=1>
          ; END_STREAM
[  0.210] send DATA frame <length=14, flags=0x01, stream_id=3>
          ; END_STREAM
[  0.210] send DATA frame <length=14, flags=0x01, stream_id=5>
          ; END_STREAM
[  0.210] send DATA frame <length=14, flags=0x01, stream_id=7>
          ; END_STREAM
[  0.210] send DATA frame <length=14, flags=0x01, stream_id=9>
          ; END_STREAM
[  0.210] recv SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):2]
          [SETTINGS_MAX_HEADER_LIST_SIZE(0x06):8192]
[  0.210] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.274] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
          [subsequent response data omitted]

All five requests were served, and five active streams were reported on the h2 connection.

To be clear, this doesn't always happen; it depends on when nghttp sees the server's SETTINGS frame. In this case, I observe the following:

[  0.011] Connected
The negotiated protocol: h2
[  0.032] recv SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):2]
          [SETTINGS_MAX_HEADER_LIST_SIZE(0x06):8192]
[  0.032] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[  0.032] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.032] send HEADERS frame <length=112, flags=0x04, stream_id=1>
          [omitted]
[  0.032] send HEADERS frame <length=14, flags=0x04, stream_id=3>
          [omitted]
[  0.032] send DATA frame <length=14, flags=0x01, stream_id=1>
          ; END_STREAM
[  0.032] send DATA frame <length=14, flags=0x01, stream_id=3>
          ; END_STREAM
[  0.035] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
          [omitted]
[  0.056] recv HEADERS frame <length=97, flags=0x24, stream_id=1>
          ; END_HEADERS | PRIORITY
          (padlen=0, dep_stream_id=0, weight=16, exclusive=0)
          ; First response header
          [omitted]
[  0.056] recv DATA frame <length=175, flags=0x00, stream_id=1>
          [omitted]
[  0.057] recv HEADERS frame <length=37, flags=0x24, stream_id=3>
          ; END_HEADERS | PRIORITY
          (padlen=0, dep_stream_id=0, weight=16, exclusive=0)
          ; First response header
          [omitted]
[  0.058] recv DATA frame <length=175, flags=0x00, stream_id=3>
[  0.232] recv GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=3, error_code=NO_ERROR(0x00), opaque_data(0)=[])
          [various payload DATA frames omitted]
[ERROR] request https://localhost:8443 failed: request HEADERS is not allowed
[ERROR] request https://localhost:8443 failed: request HEADERS is not allowed
[ERROR] request https://localhost:8443 failed: request HEADERS is not allowed
[  2.563] recv DATA frame <length=0, flags=0x01, stream_id=3>
          ; END_STREAM
[  2.563] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])
Some requests were not processed. total=5, processed=2

Note that the GOAWAY frame at 0.232 is sent by the server intentionally as part of some testing I'm doing, involving graceful shutdown of connections.

Steps to reproduce

Use nghttp (I used version 1.31.1) with the flags -v --no-dep -m M, where M is significantly greater than the server's maxConcurrentStreams, to send many requests to a Netty h2 server backed by Http2MultiplexCodec. Several invocations may be necessary to trigger the race condition, where nghttp fires off requests before seeing the server's SETTINGS.

Minimal yet complete reproducer code (or URL to code)

Will supply on request.

Netty version

4.1.24

JVM version (e.g. java -version)

java version "10" 2018-03-20
Java(TM) SE Runtime Environment 18.3 (build 10+46)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10+46, mixed mode)

OS version (e.g. uname -a)

Darwin f40f243a5fab.ant.amazon.com 16.7.0 Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64 x86_64

Start by reproducing the race with a Netty h2 server backed by Http2MultiplexCodec and nghttp using -v --no-dep -m M, where M exceeds maxConcurrentStreams. Inspect how the codec handles streams created before the peer's SETTINGS frame is observed. Done means numActiveStreams() never exceeds the configured limit and excess streams receive RST_STREAM errors.