apache / apache/druid

Netty error: "exception was thrown by a user handler while handling an exception"

Open
#5,326 5 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 31m
Merged PRs (30d)
209

Description

I think I haven't seen this before. Noticed it with the current master. I'm not sure what is going on, but it looks related to the read-timeout handler we add in NettyHttpClient.

```
2018-02-02T08:13:37,626 WARN [HttpClient-Netty-Worker-0] org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xdf1578fe, /1
72.31.12.65:39006 :> ip-172-31-3-17.ec2.internal/172.31.3.17:8304] EXCEPTION: org.jboss.netty.handler.timeout.ReadTimeoutException)
java.util.NoSuchElementException: read-timeout
at org.jboss.netty.channel.DefaultChannelPipeline.getContextOrDie(DefaultChannelPipeline.java:684) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:153) [netty-3.10.6.Final.jar:?]
at io.druid.java.util.http.client.NettyHttpClient$1.removeHandlers(NettyHttpClient.java:329) ~[java-util-0.12.0-SNAPSHOT.jar:0.12.0-SNAPSHOT]
at io.druid.java.util.http.client.NettyHttpClient$1.exceptionCaught(NettyHttpClient.java:293) ~[java-util-0.12.0-SNAPSHOT.jar:0.12.0-SNAPSHOT]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:536) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:236) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.timeout.ReadTimeoutHandler$ReadTimeoutTask$1.run(ReadTimeoutHandler.java:276) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
```

Contributor guide

Open the contributing guide

Research direction

Start in io/druid/java/util/http/client/NettyHttpClient.java at exceptionCaught around line 293 and removeHandlers around line 329, then trace the read-timeout handler behavior shown in the stack trace. Reproduce or investigate the read-timeout path; done means the exception event no longer triggers the reported NoSuchElementException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.