eclipse-ee4j / eclipse-ee4j/jersey

when using Jersey with Tomcat an AsyncResponse resume can cause a livelock issue when compression is enabled in tomcat

Open
#6,089 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
730
Forks
382
PR merge metrics
No merged PRs in 30d

Description

We are using Jersey 3.1.10 with Tomcat 10.1.54. We have compression enabled in Tomcat, and we have an async jaxrs service (using `@Suspended AsyncResponse`).

We have noticed that one of our instances of our software stack is showing that 2 threads are in a possible livelock situation over a shared `GZIPOutputStream` / `Deflater`.

These are the stacktraces of the 2 threads:
```
"client-response-8" #224 daemon prio=5 os_prio=0
cpu=612621947.84ms elapsed=709472.39s
java.lang.Thread.State: RUNNABLE
at java.util.zip.Deflater.deflate(Native Method) <-- burning CPU here
at java.util.zip.DeflaterOutputStream.deflate(...)
at java.util.zip.DeflaterOutputStream.write(...)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:148)
- locked <0x000000008ce01a68> (a java.util.zip.GZIPOutputStream)
at org.apache.coyote.http11.filters.GzipOutputFilter.doWrite(...)
at org.apache.coyote.http11.Http11OutputBuffer.doWrite(...)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(...)
at java.io.ByteArrayOutputStream.writeTo(...)
- locked <0x000000008ce02268> (a java.io.ByteArrayOutputStream)
at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(...)
at org.glassfish.jersey.message.internal.CommittingOutputStream.close(...)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(...)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(...)
at com.acme.cloud.platform.ui.framework.server.jaxrs.AsyncResponseWrapper.resume(:46)
at com.acme.cloud.platform.ui.framework.server.jaxrs.AsyncResponseWithCleanup.resume(:45)
at com.acme.cloud.platform.ui.framework.server.jaxrs.EventServiceImpl.maybeDeliverBatchResponse(:300)
at com.acme.cloud.platform.ui.framework.server.jaxrs.EventServiceImpl.lambda$handleSingleRegularResponse$10(:245)

"http-nio-8080-exec-1" #73 daemon prio=5 os_prio=0
cpu=614193943.88ms elapsed=759622.22s
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.zip.Deflater.finished(Deflater.java:442)
- locked <0x000000008ce01a50> (a java.util.zip.Deflater$DeflaterZStreamRef)
at java.util.zip.GZIPOutputStream.finish(GZIPOutputStream.java:162)
at org.apache.coyote.http11.filters.GzipOutputFilter.end(...)
at org.apache.coyote.http11.Http11OutputBuffer.end(...)
at org.apache.coyote.http11.Http11Processor.prepareResponse(...)
at org.apache.coyote.http11.Http11Processor.action(...)
at org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(...)
at org.apache.coyote.AbstractProcessor.dispatch(...)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(...)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(...)
at org.apache.tomcat.util.net.SocketProcessorBase.run(...)
```

These 2 threads have been running very actively at 80-85% of CPU during the last 8 days, while we barely have any activity in our application (due to a routing issue, most traffic for our application is being directed to other instances). So, from that I assume that these 2 threads are somehow clashing over a shared resource.

I found a similar bug report in the OpenJDK bug tracker:
https://bugs.openjdk.org/browse/JDK-8327687

This bug was closed on the JDK because it is bad usage of the API, the `GZipOutputStream` is not threadsafe.

According to the Claude 4.6 Thinking model (don't take it as definite proof):
The two lock addresses observed in the dump are separated by exactly 24 bytes on the heap:
```
0x000000008ce01a50 | Deflater$DeflaterZStreamRef | http-nio-8080-exec-1 holds
0x000000008ce01a68 | java.util.zip.GZIPOutputStream | client-response-8 holds
```

Objects 24 bytes apart on the heap are consistent with sequential allocation during `ZIPOutputStream` construction (`GZIPOutputStream → Deflater → DeflaterZStreamRef` allocated in immediate succession). Both locks belong to the same `GZIPOutputStream` instance

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.