Large number of PoolThreadCache garbage objects

We are using grpc-java version 1.16.0 in Apache Ratis for RPC communication. Apache Ratis is a java implementation of Raft consensus protocol and uses grpc for server-server and client-server communication. In the scenario there are ~10KB requests coming from the client to the server and the server is replicating those requests to other servers. There are thousands of these requests coming gradually to the server from the client. There are other small sized metadata related communications as well.

We are currently seeing a lot of GCs getting triggered and lot of garbage PoolThreadCache objects. Here is a snippet of the major unreachable heap objects.

org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache self 126Kb (< 0.1%), 2,021 object(s)
	.tinySubPageDirectCaches ↘ 173,584Kb (24.5%), 2,021 reference(s)
		org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$MemoryRegionCache[] self 284Kb (< 0.1%), 2,021 object(s)
			{array elements} ↘ 173,300Kb (24.4%), 64,672 reference(s)
				org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache self 2,021Kb (0.3%), 64,672 object(s)
					.queue ↘ 171,279Kb (24.1%), 64,672 reference(s)
						org.apache.ratis.thirdparty.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue self 40,925Kb (5.8%), 64,672 object(s)
							.buffer ↘ 130,354Kb (18.4%), 64,672 reference(s)
								Object[] self 130,354Kb (18.4%), 64,672 object(s)

	.tinySubPageHeapCaches ↘ 173,584Kb (24.5%), 2,021 reference(s)
		org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$MemoryRegionCache[] self 284Kb (< 0.1%), 2,021 object(s)
			{array elements} ↘ 173,300Kb (24.4%), 64,672 reference(s)
				org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache self 2,021Kb (0.3%), 64,672 object(s)
					.queue ↘ 171,279Kb (24.1%), 64,672 reference(s)
						org.apache.ratis.thirdparty.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue self 40,925Kb (5.8%), 64,672 object(s)
							.buffer ↘ 130,354Kb (18.4%), 64,672 reference(s)
								Object[] self 130,354Kb (18.4%), 64,672 object(s)

	.smallSubPageDirectCaches ↘ 13,641Kb (1.9%), 2,021 reference(s)
		org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$MemoryRegionCache[] self 63Kb (< 0.1%), 2,021 object(s)
			{array elements} ↘ 13,578Kb (1.9%), 8,084 reference(s)
				org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache self 252Kb (< 0.1%), 8,084 object(s)
					.queue ↘ 13,325Kb (1.9%), 8,084 reference(s)
						org.apache.ratis.thirdparty.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue self 5,115Kb (0.7%), 8,084 object(s)
							.buffer ↘ 8,210Kb (1.2%), 8,084 reference(s)
								Object[] self 8,210Kb (1.2%), 8,084 object(s)

	.smallSubPageHeapCaches ↘ 13,641Kb (1.9%), 2,021 reference(s)
		org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$MemoryRegionCache[] self 63Kb (< 0.1%), 2,021 object(s)
			{array elements} ↘ 13,578Kb (1.9%), 8,084 reference(s)
				org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache self 252Kb (< 0.1%), 8,084 object(s)
					.queue ↘ 13,325Kb (1.9%), 8,084 reference(s)
						org.apache.ratis.thirdparty.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue self 5,115Kb (0.7%), 8,084 object(s)
							.buffer ↘ 8,210Kb (1.2%), 8,084 reference(s)
								Object[] self 8,210Kb (1.2%), 8,084 object(s)

	.normalDirectCaches ↘ 5,699Kb (0.8%), 2,021 reference(s)
	
	.normalHeapCaches ↘ 5,699Kb (0.8%), 2,021 reference(s)

	.freed ↘ 31Kb (< 0.1%), 2,021 reference(s)

It seems like there is lot of garbage being generated via thread caches.