Netty IllegalReferenceCountException: refCnt: 0, decrement: 1 after enabling OpenTelemetry agent with Spring Cloud Gateway

Describe the bug

After enabling the OpenTelemetry Java agent in our Spring Cloud Gateway application (with Reactor Netty), we started seeing frequent IllegalReferenceCountException: refCnt: 0, decrement: 1 errors in the logs. This did not occur before enabling the agent.

The error stack trace is as follows:

io.netty.handler.codec.EncoderException: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:104) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.write(CombinedChannelDuplexHandler.java:526) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.write(ChannelOutboundHandlerAdapter.java:113) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.opentelemetry.javaagent.shaded.instrumentation.netty.v4_1.internal.client.HttpClientRequestTracingHandler.write(HttpClientRequestTracingHandler.java:40) [javaagent.jar:?] at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) [netty-transport-4.1.65.Final.jar:4.1.65.Final] at reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:317) [reactor-netty-0.9.20.RELEASE.jar:0.9.20.RELEASE] at reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:419) [reactor-netty-0.9.20.RELEASE.jar:0.9.20.RELEASE] at reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:219) [reactor-netty-0.9.20.RELEASE.jar:0.9.20.RELEASE] at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:90) [spring-cloud-sleuth-core-2.2.8.RELEASE.jar:2.2.8.RELEASE] at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onNext(TracingSubscriber.java:68) [javaagent.jar:?] at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:127) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:90) [spring-cloud-sleuth-core-2.2.8.RELEASE.jar:2.2.8.RELEASE] at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onNext(TracingSubscriber.java:68) [javaagent.jar:?] at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:127) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:90) [spring-cloud-sleuth-core-2.2.8.RELEASE.jar:2.2.8.RELEASE] at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onNext(TracingSubscriber.java:68) [javaagent.jar:?] at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:127) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1812) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.MonoSupplier.subscribe(MonoSupplier.java:61) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.Flux.subscribe(Flux.java:8361) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102) [reactor-netty-0.9.20.RELEASE.jar:0.9.20.RELEASE] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:234) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:201) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.Operators.complete(Operators.java:135) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:130) [reactor-netty-0.9.20.RELEASE.jar:0.9.20.RELEASE] at reactor.core.publisher.Mono.subscribe(Mono.java:4252) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:253) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.Mono.subscribe(Mono.java:4252) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336) [reactor-netty-0.9.20.RELEASE.jar:0.9.20.RELEASE] at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:65) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) [reactor-core-3.3.17.RELEASE.jar:3.3.17.RELEASE] at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:473) [reactor-netty-0.9.20.RELEASE.jar:0.9.20.RELEASE] at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:537) [reactor-netty-0.9.20.RELEASE.jar:0.9.20.RELEASE] at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.run(PooledConnectionProvider.java:635) [reactor-netty-0.9.20.RELEASE.jar:0.9.20.RELEASE] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [netty-transport-native-epoll-4.1.65.Final-linux-x86_64.jar:4.1.65.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.65.Final.jar:4.1.65.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) ~[netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) ~[netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100) ~[netty-buffer-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88) ~[netty-common-4.1.65.Final.jar:4.1.65.Final] at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:91) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final] ... 53 more

Steps to reproduce

1. Deploy a Spring Cloud Gateway application using Reactor Netty.
2. Attach the OpenTelemetry Java agent at startup.
3. Send HTTP traffic through the gateway.
4. Observe the logs for IllegalReferenceCountException.

Expected behavior

No Netty buffer reference count exceptions should occur after enabling OpenTelemetry agent.

Actual behavior

After enabling the OpenTelemetry Java agent, our application started logging frequent warnings and errors related to Netty buffer reference counting. Specifically, we see IllegalReferenceCountException: refCnt: 0, decrement: 1 exceptions in the logs during HTTP request processing. This did not happen before attaching the agent.

In addition, after enabling the agent, we have experienced container OutOfMemory (OOM) events, which did not occur previously. These memory issues seem to be correlated with the reference counting errors.

Javaagent or library instrumentation version

2.18.1

Environment

JDK:1.8.0_292
Spring Cloud Gateway: 1.0.19
Reactor Netty: 0.9.20.RELEASE
Netty: 4.1.65.Final

OS:Linux

Additional context

• This issue does not occur when the OpenTelemetry agent is not attached.
• We are also using Spring Cloud Sleuth for tracing.
• The application works as expected, but these warnings are frequent and concerning.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

Start with io.opentelemetry.javaagent.shaded.instrumentation.netty.v4_1.internal.client.HttpClientRequestTracingHandler.write, then reproduce the issue using Spring Cloud Gateway, Reactor Netty, and agent version 2.18.1 with the listed environment. Compare request processing with and without the agent, focusing on the IllegalReferenceCountException and reported OOM events. Done means HTTP traffic no longer produces these errors or correlated memory failures.