eclipse-ee4j / eclipse-ee4j/jersey
when using Jersey with Tomcat an AsyncResponse resume can cause a livelock issue when compression is enabled in tomcat
- Dominant language
- Java
- Stars
- 730
- Forks
- 382
- PR merge metrics
- No merged PRs in 30d
Description
We are using Jersey 3.1.10 with Tomcat 10.1.54. We have compression enabled in Tomcat, and we have an async jaxrs service (using `@Suspended AsyncResponse`).
We have noticed that one of our instances of our software stack is showing that 2 threads are in a possible livelock situation over a shared `GZIPOutputStream` / `Deflater`.
These are the stacktraces of the 2 threads:
```
"client-response-8" #224 daemon prio=5 os_prio=0
cpu=612621947.84ms elapsed=709472.39s
java.lang.Thread.State: RUNNABLE
at java.util.zip.Deflater.deflate(Native Method) <-- burning CPU here
at java.util.zip.DeflaterOutputStream.deflate(...)
at java.util.zip.DeflaterOutputStream.write(...)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:148)
- locked <0x000000008ce01a68> (a java.util.zip.GZIPOutputStream)
at org.apache.coyote.http11.filters.GzipOutputFilter.doWrite(...)
at org.apache.coyote.http11.Http11OutputBuffer.doWrite(...)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(...)
at java.io.ByteArrayOutputStream.writeTo(...)
- locked <0x000000008ce02268> (a java.io.ByteArrayOutputStream)
at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(...)
at org.glassfish.jersey.message.internal.CommittingOutputStream.close(...)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(...)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(...)
at com.acme.cloud.platform.ui.framework.server.jaxrs.AsyncResponseWrapper.resume(:46)
at com.acme.cloud.platform.ui.framework.server.jaxrs.AsyncResponseWithCleanup.resume(:45)
at com.acme.cloud.platform.ui.framework.server.jaxrs.EventServiceImpl.maybeDeliverBatchResponse(:300)
at com.acme.cloud.platform.ui.framework.server.jaxrs.EventServiceImpl.lambda$handleSingleRegularResponse$10(:245)
"http-nio-8080-exec-1" #73 daemon prio=5 os_prio=0
cpu=614193943.88ms elapsed=759622.22s
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.zip.Deflater.finished(Deflater.java:442)
- locked <0x000000008ce01a50> (a java.util.zip.Deflater$DeflaterZStreamRef)
at java.util.zip.GZIPOutputStream.finish(GZIPOutputStream.java:162)
at org.apache.coyote.http11.filters.GzipOutputFilter.end(...)
at org.apache.coyote.http11.Http11OutputBuffer.end(...)
at org.apache.coyote.http11.Http11Processor.prepareResponse(...)
at org.apache.coyote.http11.Http11Processor.action(...)
at org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(...)
at org.apache.coyote.AbstractProcessor.dispatch(...)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(...)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(...)
at org.apache.tomcat.util.net.SocketProcessorBase.run(...)
```
These 2 threads have been running very actively at 80-85% of CPU during the last 8 days, while we barely have any activity in our application (due to a routing issue, most traffic for our application is being directed to other instances). So, from that I assume that these 2 threads are somehow clashing over a shared resource.
I found a similar bug report in the OpenJDK bug tracker:
https://bugs.openjdk.org/browse/JDK-8327687
This bug was closed on the JDK because it is bad usage of the API, the `GZipOutputStream` is not threadsafe.
According to the Claude 4.6 Thinking model (don't take it as definite proof):
The two lock addresses observed in the dump are separated by exactly 24 bytes on the heap:
```
0x000000008ce01a50 | Deflater$DeflaterZStreamRef | http-nio-8080-exec-1 holds
0x000000008ce01a68 | java.util.zip.GZIPOutputStream | client-response-8 holds
```
Objects 24 bytes apart on the heap are consistent with sequential allocation during `ZIPOutputStream` construction (`GZIPOutputStream → Deflater → DeflaterZStreamRef` allocated in immediate succession). Both locks belong to the same `GZIPOutputStream` instance
Contributor guide
Assessment
This issue has not been assessed yet.