EofException caused by ClosedChannelException
- Dominant language
- Java
- Stars
- 4.1k
- Forks
- 2k
- Avg merge
- 3d 56m
- Merged PRs (30d)
- 48
Description
**Jetty version(s)**
Observed in 12.0.6-12.0.10 with ee10.
First observed in March 2024 when we deployed a release of our server that upgraded from 11.0.20 to 12.0.6+ee10. It had never been observed prior to that (at least, I can't find it in our retained error logs from the last year).
**Jetty Environment**
ee10
**Java version/vendor** `(use: java -version)`
```
openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment Corretto-21.0.3.9.1 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.3.9.1 (build 21.0.3+9-LTS, mixed mode, sharing)
```
**OS type/version**
Amazon Linux 2
**Description**
We periodically but rarely (a few times a month) see `EofException` caused by `ClosedChannelException`. We more routinely see plain `IOException` when a client disconnects without reading the response, and we ignore that error. We've considered ignoring this error too, but our understanding is that it should only happen if Jetty closed its side of the connection and then subsequently attempted to write, so it seems like this could be some kind of connection state management issue in Jetty. The relevant filters/servlets are purely synchronous: they do not use `startAsync`, and they do not use `HttpServletRequest`/`HttpServletResponse` from other threads. Recent stack trace from 12.0.10+ee10 (as logged by Logback, which writes the explicit `null` for null messages).
```
org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:117)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:368)
at org.eclipse.jetty.server.internal.HttpConnection$SendCallback.process(HttpConnection.java:842)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:250)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:231)
at org.eclipse.jetty.server.internal.HttpConnection$HttpStreamOverHTTP1.send(HttpConnection.java:1429)
at org.eclipse.jetty.server.HttpStream$Wrapper.send(HttpStream.java:179)
at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.write(HttpChannelState.java:1263)
at org.eclipse.jetty.server.Response$Wrapper.write(Response.java:751)
at org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56)
at org.eclipse.jetty.ee10.servlet.ServletContextResponse.write(ServletContextResponse.java:288)
at org.eclipse.jetty.ee10.servlet.HttpOutput.channelWrite(HttpOutput.java:205)
at org.eclipse.jetty.ee10.servlet.HttpOutput.close(HttpOutput.java:562)
redacted
at jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)
at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:824)
at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436)
at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:851)
at org.eclipse.jetty.server.Server.handle(Server.java:179)
at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:635)
at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:411)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.nio.channels.ClosedChannelException: null
at java.base/sun.nio.ch.SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:222)
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:568)
at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:660)
at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:111)
... 52 common frames omitted
```
**How to reproduce?**
I don't know. I've tried various things against a local Jetty instance (early client close, delayed client close, slow client write, slow server write, different buffered amounts to trigger an error from `.write` vs `.close`, etc.), but the only relevant error I can cause is `EofException` with `IOException: Broken pipe` as expected.
Contributor guide
Research direction
Start with SocketChannelEndPoint.flush and follow the WriteFlusher and HttpConnection send path shown in the stack trace, then inspect HttpOutput.close. Use the reported client-close and delayed-write scenarios as reproduction starting points. Done means the ClosedChannelException path is understood and covered by a regression test or a confirmed resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100