use netty as http client, out of memory when run with big payload

Our application use netty as http client, and run in kubernetes platform.
The application works fine with small payload, like less than 10K. This has been tested for 72h and no out of memory issue.

But when we test with big payload, 200tps and 1M payload, OOM occur after several hours.
We test with leak detection level paranoid, and no error was found.

We enable NMT, and the part that can track with NMT seems normal.
We have try the following, and the memory issue is gone:

• disable direct memory by -Dio.netty.allocator.numDirectArenas=0 -Dio.netty.noPreferDirect=true
• rebuild the image with native library excluded, such as netty-transport-native-epoll
• add environment variable MALLOC_ARENA_MAX=1

Any clue of what is causing the issue?
Or tools/guides to identify the root cause？
Thanks.

# jvm parameters
4Gi memory is assigned to the container, both request and limit set to 4Gi.

-XX:MinTLABSize=4096 -XX:+UnlockDiagnosticVMOptions -XX:GuaranteedSafepointInterval=0
-XX:+UseCountedLoopSafepoints -XX:-UseBiasedLocking -XX:MetaspaceSize=256m
-XX:MaxMetaspaceSize=512m -XX:+PrintFlagsFinal -XX:+PrintCommandLineFlags
-XX:-OmitStackTraceInFastThrow -XX:+CrashOnOutOfMemoryError -XX:+DebugNonSafepoints
-XX:InitialRAMPercentage=70.000000 -XX:MinRAMPercentage=50.000000 -XX:MaxRAMPercentage=70.000000

# We use PooledByteBufAllocator.DEFAULT allocator
content = PooledByteBufAllocator.DEFAULT.buffer(available);


Native Memory Tracking:

Total: reserved=4425712KB, committed=3418588KB
-                 Java Heap (reserved=2936832KB, committed=2936832KB)
                            (mmap: reserved=2936832KB, committed=2936832KB)

-                     Class (reserved=608747KB, committed=102123KB)
                            (classes #15939)
                            (  instance classes #14880, array classes #1059)
                            (malloc=2539KB #49899)
                            (mmap: reserved=606208KB, committed=99584KB)
                            (  Metadata:   )
                            (    reserved=90112KB, committed=88320KB)
                            (    used=86554KB)
                            (    free=1766KB)
                            (    waste=0KB =0.00%)
                            (  Class space:)
                            (    reserved=516096KB, committed=11264KB)
                            (    used=10223KB)
                            (    free=1041KB)
                            (    waste=0KB =0.00%)

-                    Thread (reserved=369000KB, committed=48392KB)
                            (thread #358)
                            (stack: reserved=367992KB, committed=47384KB)
                            (malloc=590KB #2150)
                            (arena=418KB #714)

-                      Code (reserved=251970KB, committed=72078KB)
                            (malloc=4282KB #17460)
                            (mmap: reserved=247688KB, committed=67796KB)

-                        GC (reserved=161090KB, committed=161090KB)
                            (malloc=18954KB #50940)
                            (mmap: reserved=142136KB, committed=142136KB)

-                  Compiler (reserved=2585KB, committed=2585KB)
                            (malloc=2453KB #1877)
                            (arena=133KB #5)

-                  Internal (reserved=27618KB, committed=27618KB)
                            (malloc=27586KB #5724)
                            (mmap: reserved=32KB, committed=32KB)

-                     Other (reserved=40386KB, committed=40386KB)
                            (malloc=40386KB #4057)

-                    Symbol (reserved=20344KB, committed=20344KB)
                            (malloc=17332KB #220741)
                            (arena=3013KB #1)

-    Native Memory Tracking (reserved=6320KB, committed=6320KB)
                            (malloc=575KB #8136)
                            (tracking overhead=5745KB)

-               Arena Chunk (reserved=188KB, committed=188KB)
                            (malloc=188KB)

-                   Logging (reserved=4KB, committed=4KB)
                            (malloc=4KB #189)

-                 Arguments (reserved=21KB, committed=21KB)
                            (malloc=21KB #576)

-                    Module (reserved=262KB, committed=262KB)
                            (malloc=262KB #2383)

-              Synchronizer (reserved=336KB, committed=336KB)
                            (malloc=336KB #2724)

-                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)



Attaching to process ID 14, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.18+0-suse-150000.3.93.1-x8664

using thread-local object allocation.
Garbage-First (G1) GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 3007315968 (2868.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 1803550720 (1720.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 268435456 (256.0MB)
   CompressedClassSpaceSize = 528482304 (504.0MB)
   MaxMetaspaceSize         = 536870912 (512.0MB)
   G1HeapRegionSize         = 1048576 (1.0MB)

Heap Usage:
G1 Heap:
   regions  = 2868
   capacity = 3007315968 (2868.0MB)
   used     = 715865928 (682.7029495239258MB)
   free     = 2291450040 (2185.297050476074MB)
   23.80414747294023% used
G1 Young Generation:
Eden Space:
   regions  = 13
   capacity = 211812352 (202.0MB)
   used     = 13631488 (13.0MB)
   free     = 198180864 (189.0MB)
   6.435643564356436% used
Survivor Space:
   regions  = 1
   capacity = 1048576 (1.0MB)
   used     = 1048576 (1.0MB)
   free     = 0 (0.0MB)
   100.0% used
G1 Old Generation:
   regions  = 673
   capacity = 2794455040 (2665.0MB)
   used     = 701185864 (668.7029495239258MB)
   free     = 2093269176 (1996.2970504760742MB)
   25.09204313410603% used


USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        14  145 12.2 8683040 4041576 ?     Sl   Nov21 1375:17 java 

Netty version

Try both netty 4.1.89 and 4.1.101

JVM version (e.g. java -version)

Try both openjdk 11.0.18 and 17.0.9
java -version
openjdk version "11.0.18" 2023-01-17
OpenJDK Runtime Environment (build 11.0.18+0-suse-150000.3.93.1-x8664)
OpenJDK 64-Bit Server VM (build 11.0.18+0-suse-150000.3.93.1-x8664, mixed mode)

OS version (e.g. uname -a)

Linux 5.14.21-150400.24.63-default #1 SMP PREEMPT_DYNAMIC Tue May 2 15:49:04 UTC 2023 (fd0cc4f) x86_64 x86_64 x86_64 GNU/Linux

Start at the PooledByteBufAllocator.DEFAULT.buffer(available) entry point and compare the direct-memory and native-transport configurations described in the report. Reproduce the 1M-payload, 200 TPS load while inspecting NMT and container memory; done means isolating the memory source with evidence or documenting a reliable diagnostic path.