clj-commons / clj-commons/aleph
io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 0000000401
- Dominant language
- Clojure
- Stars
- 2.6k
- Forks
- 242
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 1
Description
[Occasionally](https://app.circleci.com/pipelines/github/clj-commons/aleph/798/workflows/3d2f1ce2-58ac-4582-8f13-8bc9bcb04a03/jobs/790/parallel-runs/0/steps/0-103), `aleph.http-test/test-bulk-requests` runs into the following error:
```
845613 tid=49 ERROR common - Exception in channel (conn-ex-handler).
io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 0000000401
at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:107)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:351)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:245)
at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1519)
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1390)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1430)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at manifold.executor$thread_factory$reify__12504$f__12505.invoke(executor.clj:71)
at clojure.lang.AFn.run(AFn.java:22)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
```
The previous line logged for `tid=49` was this, indicating that it's the client who ran into the error and thus the server is at fault for sending the invalid response:
```
845610 tid=49 DEBUG aleph-client - [id: 0x6f360c1c, L:/127.0.0.1:44312 - R:localhost/127.0.0.1:8082](H2 - 3) FLUSH
```
The first 5 bytes cited in the error message (`0000000401`) indicate a `SETTINGS` frame but with the `ACK` bit set which [Netty explicitly rejects](https://github.com/netty/netty/commit/8f334885ef58aa0dda4107391200c29527106df7) as it's in violation of the protocol.
Contributor guide
Research direction
Start with aleph.http-test/test-bulk-requests and inspect the HTTP/2 client and server setup around that test. Reproduce the intermittent failure in the CircleCI run or locally, then trace why the client receives a SETTINGS frame with the ACK bit set. Done means the test no longer encounters the invalid first frame and the relevant HTTP/2 behavior is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clojure
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100