eclipse-ee4j / eclipse-ee4j/jersey

Jersey with Tomcat leaks connection when AsyncResponse resume() fails to flush out headers.

Open
#4,027 0 comments 0 reactions 0 assignees View on GitHub
async
Dominant language
Java
Stars
730
Forks
382
PR merge metrics
No merged PRs in 30d

Description

The root cause appears to be that the AsyncContext does not get closed. Proximate causes: `ContainerResponse.close()` does not call `requestContext.getResponseWriter().commit()` because `messageContext.close()` throws because `commitingOutputStream.close()` throws.

Sanitized stack traces that we actually see:
```

2019-01-14 10:32:33.005 ERROR 149715 --- [onse-executor-0] o.g.j.server.ServerRuntime$Responder : An I/O error has occurred while writing a response message entity to the container output stream.
--
 
org.glassfish.jersey.server.internal.process.MappableException: com.fasterxml.jackson.databind.JsonMappingException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers. (through reference chain: REDACTED)

at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92)
--
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:934)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:966)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:922)
at com.pany.code.ThatCalls.resume(ThatCalls.java:123)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers. (through reference chain: REDACTED)

at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:388)
--
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:348)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:343)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:697)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292)
at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429)
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951)
at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86)
... 19 common frames omitted
Caused by: org.apache.coyote.http11.HeadersTooLargeException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.
--
at org.apache.coyote.http11.Http11OutputBuffer.checkLengthBeforeWrite(Http11OutputBuffer.java:543)
at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:468)
at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:454)
at org.apache.coyote.http11.Http11OutputBuffer.sendHeader(Http11OutputBuffer.java:414)
at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1388)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:346)
at org.apache.coyote.Response.action(Response.java:173)
at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:219)
at org.apache.coyote.Response.doWrite(Response.java:541)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:825)
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:730)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:391)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:369)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:325)
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:233)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:503)
at com.fasterxml.jackson.databind.ser.std.EnumSerializer.serialize(EnumSerializer.java:141)
at com.fasterxml.jackson.databind.ser.std.EnumSerializer.serialize(EnumSerializer.java:27)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689)
... 36 common frames omitted

2019-01-14 10:32:33.010 ERROR 149715 --- [onse-executor-0] o.g.j.server.ServerRuntime$Responder : Error while closing the output stream in order to commit response.
--
 
org.apache.coyote.http11.HeadersTooLargeException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.
at org.apache.coyote.http11.Http11OutputBuffer.checkLengthBeforeWrite(Http11OutputBuffer.java:543)
at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:468)
at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:454)
at org.apache.coyote.http11.Http11OutputBuffer.sendHeader(Http11OutputBuffer.java:414)
at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1388)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:346)
at org.apache.coyote.Response.action(Response.java:175)
at org.apache.coyote.Response.sendHeaders(Response.java:357)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:303)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:284)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.flush(ResponseWriter.java:330)
at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:292)
at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:876)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:412)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:784)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:934)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:966)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:922)
at com.pany.code.ThatCalls.resume(ThatCalls.java:123)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```

Contributor guide

Open the contributing guide

Research direction

Start with ServerRuntime$AsyncResponder.resume and follow the failure through ServerRuntime$Responder.writeResponse, ContainerResponse.close, OutboundMessageContext.close, and CommittingOutputStream. Reproduce the oversized-header failure described in the stack traces, then determine how the async context should be closed when flushing or closing the response fails and add coverage for the observed leak.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.