clj-commons / clj-commons/aleph

io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 0000000401

Open
#772 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Clojure
Stars
2.6k
Forks
242
Avg merge
4d 5h
Merged PRs (30d)
1

Description

[Occasionally](https://app.circleci.com/pipelines/github/clj-commons/aleph/798/workflows/3d2f1ce2-58ac-4582-8f13-8bc9bcb04a03/jobs/790/parallel-runs/0/steps/0-103), `aleph.http-test/test-bulk-requests` runs into the following error:

```
845613 tid=49 ERROR common - Exception in channel (conn-ex-handler).
io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 0000000401
at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:107)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:351)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:245)
at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1519)
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1390)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1430)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at manifold.executor$thread_factory$reify__12504$f__12505.invoke(executor.clj:71)
at clojure.lang.AFn.run(AFn.java:22)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
```

The previous line logged for `tid=49` was this, indicating that it's the client who ran into the error and thus the server is at fault for sending the invalid response:

```
845610 tid=49 DEBUG aleph-client - [id: 0x6f360c1c, L:/127.0.0.1:44312 - R:localhost/127.0.0.1:8082](H2 - 3) FLUSH
```

The first 5 bytes cited in the error message (`0000000401`) indicate a `SETTINGS` frame but with the `ACK` bit set which [Netty explicitly rejects](https://github.com/netty/netty/commit/8f334885ef58aa0dda4107391200c29527106df7) as it's in violation of the protocol.

Contributor guide

Open the contributing guide

Research direction

Start with aleph.http-test/test-bulk-requests and inspect the HTTP/2 client and server setup around that test. Reproduce the intermittent failure in the CircleCI run or locally, then trace why the client receives a SETTINGS frame with the ACK bit set. Done means the test no longer encounters the invalid first frame and the relevant HTTP/2 behavior is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.