eclipse-ee4j / eclipse-ee4j/jersey
Jersey with Tomcat leaks connection when AsyncResponse resume() fails to flush out headers.
- Dominant language
- Java
- Stars
- 730
- Forks
- 382
- PR merge metrics
- No merged PRs in 30d
Description
The root cause appears to be that the AsyncContext does not get closed. Proximate causes: `ContainerResponse.close()` does not call `requestContext.getResponseWriter().commit()` because `messageContext.close()` throws because `commitingOutputStream.close()` throws.
Sanitized stack traces that we actually see:
```
2019-01-14 10:32:33.005 ERROR 149715 --- [onse-executor-0] o.g.j.server.ServerRuntime$Responder : An I/O error has occurred while writing a response message entity to the container output stream.
--
org.glassfish.jersey.server.internal.process.MappableException: com.fasterxml.jackson.databind.JsonMappingException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers. (through reference chain: REDACTED)
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92)
--
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:934)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:966)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:922)
at com.pany.code.ThatCalls.resume(ThatCalls.java:123)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers. (through reference chain: REDACTED)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:388)
--
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:348)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:343)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:697)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292)
at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429)
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951)
at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86)
... 19 common frames omitted
Caused by: org.apache.coyote.http11.HeadersTooLargeException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.
--
at org.apache.coyote.http11.Http11OutputBuffer.checkLengthBeforeWrite(Http11OutputBuffer.java:543)
at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:468)
at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:454)
at org.apache.coyote.http11.Http11OutputBuffer.sendHeader(Http11OutputBuffer.java:414)
at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1388)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:346)
at org.apache.coyote.Response.action(Response.java:173)
at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:219)
at org.apache.coyote.Response.doWrite(Response.java:541)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:825)
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:730)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:391)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:369)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:325)
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:233)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:503)
at com.fasterxml.jackson.databind.ser.std.EnumSerializer.serialize(EnumSerializer.java:141)
at com.fasterxml.jackson.databind.ser.std.EnumSerializer.serialize(EnumSerializer.java:27)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689)
... 36 common frames omitted
2019-01-14 10:32:33.010 ERROR 149715 --- [onse-executor-0] o.g.j.server.ServerRuntime$Responder : Error while closing the output stream in order to commit response.
--
org.apache.coyote.http11.HeadersTooLargeException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.
at org.apache.coyote.http11.Http11OutputBuffer.checkLengthBeforeWrite(Http11OutputBuffer.java:543)
at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:468)
at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:454)
at org.apache.coyote.http11.Http11OutputBuffer.sendHeader(Http11OutputBuffer.java:414)
at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1388)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:346)
at org.apache.coyote.Response.action(Response.java:175)
at org.apache.coyote.Response.sendHeaders(Response.java:357)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:303)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:284)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.flush(ResponseWriter.java:330)
at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:292)
at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:876)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:412)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:784)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:934)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:966)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:922)
at com.pany.code.ThatCalls.resume(ThatCalls.java:123)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
Contributor guide
Research direction
Start with ServerRuntime$AsyncResponder.resume and follow the failure through ServerRuntime$Responder.writeResponse, ContainerResponse.close, OutboundMessageContext.close, and CommittingOutputStream. Reproduce the oversized-header failure described in the stack traces, then determine how the async context should be closed when flushing or closing the response fails and add coverage for the observed leak.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100