jetty / jetty/jetty.project

EofException caused by ClosedChannelException

Open
#11,937 6 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
4.1k
Forks
2k
Avg merge
3d 56m
Merged PRs (30d)
48

Description

**Jetty version(s)**
Observed in 12.0.6-12.0.10 with ee10.

First observed in March 2024 when we deployed a release of our server that upgraded from 11.0.20 to 12.0.6+ee10. It had never been observed prior to that (at least, I can't find it in our retained error logs from the last year).

**Jetty Environment**
ee10

**Java version/vendor** `(use: java -version)`
```
openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment Corretto-21.0.3.9.1 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.3.9.1 (build 21.0.3+9-LTS, mixed mode, sharing)
```

**OS type/version**
Amazon Linux 2

**Description**
We periodically but rarely (a few times a month) see `EofException` caused by `ClosedChannelException`. We more routinely see plain `IOException` when a client disconnects without reading the response, and we ignore that error. We've considered ignoring this error too, but our understanding is that it should only happen if Jetty closed its side of the connection and then subsequently attempted to write, so it seems like this could be some kind of connection state management issue in Jetty. The relevant filters/servlets are purely synchronous: they do not use `startAsync`, and they do not use `HttpServletRequest`/`HttpServletResponse` from other threads. Recent stack trace from 12.0.10+ee10 (as logged by Logback, which writes the explicit `null` for null messages).
```
org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:117)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:368)
at org.eclipse.jetty.server.internal.HttpConnection$SendCallback.process(HttpConnection.java:842)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:250)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:231)
at org.eclipse.jetty.server.internal.HttpConnection$HttpStreamOverHTTP1.send(HttpConnection.java:1429)
at org.eclipse.jetty.server.HttpStream$Wrapper.send(HttpStream.java:179)
at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.write(HttpChannelState.java:1263)
at org.eclipse.jetty.server.Response$Wrapper.write(Response.java:751)
at org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56)
at org.eclipse.jetty.ee10.servlet.ServletContextResponse.write(ServletContextResponse.java:288)
at org.eclipse.jetty.ee10.servlet.HttpOutput.channelWrite(HttpOutput.java:205)
at org.eclipse.jetty.ee10.servlet.HttpOutput.close(HttpOutput.java:562)
redacted
at jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)
at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:824)
at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436)
at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:851)
at org.eclipse.jetty.server.Server.handle(Server.java:179)
at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:635)
at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:411)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.nio.channels.ClosedChannelException: null
at java.base/sun.nio.ch.SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:222)
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:568)
at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:660)
at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:111)
... 52 common frames omitted

```

**How to reproduce?**
I don't know. I've tried various things against a local Jetty instance (early client close, delayed client close, slow client write, slow server write, different buffered amounts to trigger an error from `.write` vs `.close`, etc.), but the only relevant error I can cause is `EofException` with `IOException: Broken pipe` as expected.

Contributor guide

Open the contributing guide

Research direction

Start with SocketChannelEndPoint.flush and follow the WriteFlusher and HttpConnection send path shown in the stack trace, then inspect HttpOutput.close. Use the reported client-close and delayed-write scenarios as reproduction starting points. Done means the ClosedChannelException path is understood and covered by a regression test or a confirmed resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.