memory allocation attack?

Hello Waterfall community, i'm owning a smaller minecraft server with about 50 max concurrent players.
I am recently facing bot attacks where multiple ips (proxies?) connect to the waterfall proxy, each allocating 16MB direct memory and thus rendering the server unusable in seconds.

I've allocated 512MB memory to waterfall, which was plently for the last 3 years. I still doubled it to 1 gig for now, but the DOS "attack" still manages to fill the ram in seconds.

This exception is spammed to console during attack:

[00:12:21] [Netty Worker IO Thread #9/ERROR]: [/36.94.40.114:53052] <-> InitialHandler - encountered exception
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 520093703, max: 536870912)
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:775) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:730) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:645) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:621) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:204) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:188) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:138) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.buffer.PoolArena.reallocate(PoolArena.java:288) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:118) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:307) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:282) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1105) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:99) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:274) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[Waterfall.jar:git:Waterfall-Bootstrap:1.16-R0.5-SNAPSHOT:c031df1:395]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]

You could argue that this is a ddos attack and can't be fixed by bungeecord/waterfall.
However, the host machine was using about 30% cpu and 10% of it's network resources, it's really only bungeecord that is struggeling to keep up with that many requests.

I do use iptables to rate limit new connections per ip to the bc, but this does not really help as the connections come from too many different ips (proxy list?). I now added a global rate limit for syn packets to bungeecord, which somewhat mitigates the attack by not crashin the server. However, no new players can join while an attack is running. So this is not a permanent option :/

I also don't make profit from my server, so i can't afford professional layer 7 ddos mitigation. Hoping to get help here is my only option.
Any help is appreciated.

Start at the InitialHandler path in the stack trace and follow Netty's ByteToMessageDecoder handling of inbound data and direct-memory growth. Reproduce the connection pattern described, then establish a bounded-memory behavior that still permits legitimate players to join and verify it with an attack simulation.