'SslClosedEngineException: SSLEngine closed already' under heavy load

Expected behavior

No exception occur when writing to the Channel with SSLHandler and lots of concurrent connections (200+)

Actual behavior

SslClosedEngineException is thrown when writing to a channel when there are a lot of concurrent connections.

It only happens when SSLHandler is used. Without one, no errors take place.

Steps to reproduce

1. Initialize Netty
2. Run a loop that does the following hundreds of times a second:

• Open new connection with a SSLHandler
• Wait until connected, then write some bytes to a channel
• Close the channel 1000ms after the connection

3. Wait until an exception occurrs

Minimal yet complete reproducer code (or URL to code)

Client.java

	public static void main(String[] args) throws Exception {
		NioEventLoopGroup group = new NioEventLoopGroup();
		SslContext ctx = SslContextBuilder
				.forClient()
				.trustManager(InsecureTrustManagerFactory.INSTANCE)
				.protocols("TLSv1.2")
				.build();

		Bootstrap bootstrap = new Bootstrap();
		bootstrap.group(new NioEventLoopGroup()).channel(NioSocketChannel.class)
				.handler(new ChannelInitializer<SocketChannel>() {
					@Override
					protected void initChannel(SocketChannel ch) {
						ChannelPipeline pipeline = ch.pipeline();
						final SslHandler sslHandler = ctx.newHandler(ch.alloc());
						pipeline.addLast(new PreHandler());
						pipeline.addLast("ssl", sslHandler);
						pipeline.addLast("length-decoder", new LengthFieldBasedFrameDecoder(MAX_VALUE, 0, 4, 0, 4));
						pipeline.addLast("bytearray-decoder", new ByteArrayDecoder());
						pipeline.addLast("length-encoder", new LengthFieldPrepender(4));
						pipeline.addLast("bytearray-encoder", new ByteArrayEncoder());
						BlockingByteArrayClientHandler handler = new BlockingByteArrayClientHandler();
						pipeline.addLast("handler", handler);
					}
				});

		while (true) {
			ChannelFuture future = bootstrap.connect("localhost", 8443);
			future.addListener((ChannelFutureListener)connFuture -> {
				if (connFuture.isSuccess()) {
					Channel ch = connFuture.channel();
					byte[] packet = new byte[256];
					new Random().nextBytes(packet);
					ch.writeAndFlush(packet).addListener(writeFuture -> {
						if (!writeFuture.isSuccess()) {
							System.out.println("Write failed: " + writeFuture.cause());
						}
					});
					group.schedule(() -> {
						ch.close();
					}, 1000L, TimeUnit.MILLISECONDS);
				} else {
					System.out.println("conn failed " + connFuture.cause());
				}
			});
			if (ThreadLocalRandom.current().nextBoolean()){
				Thread.sleep(1);
			}

		}

Server.java

    public static void main(String[] args) throws Exception {
        EventLoopGroup bossGroup = new NioEventLoopGroup(1);
        EventLoopGroup workerGroup = new NioEventLoopGroup();
        try {
            ServerBootstrap b = new ServerBootstrap();
            b.group(bossGroup, workerGroup)
                    .channel(NioServerSocketChannel.class)
                    .childHandler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        protected void initChannel(SocketChannel channel) throws Exception {
                            ChannelPipeline pipeline = channel.pipeline();
                            SSLEngine engine =
                                    SecureSocketSslContextFactory.getServerContext().createSSLEngine();
                            engine.setUseClientMode(false);
                            pipeline.addLast("ssl", new SslHandler(engine));
                            pipeline.addLast("length-decoder", new LengthFieldBasedFrameDecoder(Integer.MAX_VALUE, 0, 4, 0, 4));
                            pipeline.addLast("bytearray-decoder", new ByteArrayDecoder());
                            pipeline.addLast("length-encoder", new LengthFieldPrepender(4));
                            pipeline.addLast("bytearray-encoder", new ByteArrayEncoder());
                            pipeline.addLast("handler", new SecureSocketServerhandler2());
                        }
                    });

            b.bind(8443).sync().channel().closeFuture().sync();
        } finally {
            bossGroup.shutdownGracefully();
            workerGroup.shutdownGracefully();
        }
    }

Netty version

4.1.78.Final

JVM version (e.g. java -version)

$ java -version
java version "11.0.8" 2020-07-14 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.8+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.8+10-LTS, mixed mode)

OS version (e.g. uname -a)

$ uname -a
MINGW64_NT-10.0-19044 3.1.4-340.x86_64 2020-05-19 12:55 UTC x86_64 Msys

Same problem on Linux:

user@host:~> uname -a
Linux host 5.3.18-150300.59.68-default #1 SMP Wed May 4 11:29:09 UTC 2022 (ea30951) x86_64 x86_64 x86_64 GNU/Linux

Debug output

17:57:13.050 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
17:57:13.080 [main] DEBUG i.n.c.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 8
17:57:13.114 [main] DEBUG i.n.u.i.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
17:57:13.114 [main] DEBUG i.n.u.i.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
17:57:13.181 [main] DEBUG i.n.util.internal.PlatformDependent0 - -Dio.netty.noUnsafe: false
17:57:13.181 [main] DEBUG i.n.util.internal.PlatformDependent0 - Java version: 8
17:57:13.186 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
17:57:13.191 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
17:57:13.196 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.storeFence: available
17:57:13.198 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
17:57:13.199 [main] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
17:57:13.200 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
17:57:13.200 [main] DEBUG i.n.util.internal.PlatformDependent0 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
17:57:13.200 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.(long, int): available
17:57:13.201 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
17:57:13.208 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: C:\Users\U_M15C5\AppData\Local\Temp (java.io.tmpdir)
17:57:13.209 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
17:57:13.210 [main] DEBUG i.n.util.internal.PlatformDependent - Platform: Windows
17:57:13.211 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.maxDirectMemory: 1908932608 bytes
17:57:13.211 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.uninitializedArrayAllocationThreshold: -1
17:57:13.212 [main] DEBUG io.netty.util.internal.CleanerJava6 - java.nio.ByteBuffer.cleaner(): available
17:57:13.212 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
17:57:13.214 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
17:57:13.214 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
17:57:13.227 [main] DEBUG i.n.util.internal.PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available
17:57:13.492 [main] DEBUG io.netty.handler.ssl.OpenSsl - netty-tcnative not in the classpath; OpenSslEngine will be unavailable.
17:57:13.924 [main] DEBUG io.netty.handler.ssl.JdkSslContext - Default protocols (JDK): [TLSv1.2, TLSv1.1, TLSv1]
17:57:13.924 [main] DEBUG io.netty.handler.ssl.JdkSslContext - Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA]
17:57:13.960 [main] DEBUG io.netty.channel.DefaultChannelId - -Dio.netty.processId: 16784 (auto-detected)
17:57:13.962 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv4Stack: false
17:57:13.962 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv6Addresses: false
17:57:14.067 [main] DEBUG io.netty.util.NetUtilInitializations - Loopback interface: lo (Software Loopback Interface 1, 127.0.0.1)
17:57:14.068 [main] DEBUG io.netty.util.NetUtil - Failed to get SOMAXCONN from sysctl and file \proc\sys\net\core\somaxconn. Default: 200
17:57:14.171 [main] DEBUG io.netty.channel.DefaultChannelId - -Dio.netty.machineId: 00:50:56:ff:fe:ba:4a:68 (auto-detected)
17:57:14.184 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
17:57:14.184 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.targetRecords: 4
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 9
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 4194304
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimIntervalMillis: 0
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.useCacheForAllThreads: false
17:57:14.218 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
17:57:14.231 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: pooled
17:57:14.231 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 0
17:57:14.231 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.maxThreadLocalCharBufferSize: 16384
17:57:14.247 [main] DEBUG i.n.util.internal.ThreadLocalRandom - -Dio.netty.initialSeedUniquifier: 0x739012402ab811cc
17:57:14.311 [nioEventLoopGroup-3-3] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkAccessible: true
17:57:14.311 [nioEventLoopGroup-3-3] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkBounds: true
17:57:14.312 [nioEventLoopGroup-3-3] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@43e108cb
17:57:14.315 [nioEventLoopGroup-3-3] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacityPerThread: 4096
17:57:14.315 [nioEventLoopGroup-3-3] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.ratio: 8
17:57:14.315 [nioEventLoopGroup-3-3] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.chunkSize: 32
17:57:14.315 [nioEventLoopGroup-3-3] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.blocking: false

Start by running the Client.java and Server.java reproducer with Netty 4.1.78.Final, focusing on the SslHandler setup and the concurrent connect, write, and close sequence. Confirm whether SslClosedEngineException appears under sustained load, then trace the SSL handling path to identify the failure. Done means the reproducer completes without the exception under the reported load, with regression coverage if an existing test location is identified.