Memory Leak while using BES

### Description of the bug:

We extensively use bes, and we've noticed it uses a very large and increasing amount of memory which causes our builds to frequently crash. We are able to have our builds pass without memory issue when we disable our bes_backend, so we know the issue is related directly to bes.

In a recent crash I was able to capture a memory snapshot, and the BES service appears to be holding on to several million protobuf refs (image), this appears to eat into the direct buffers that are needed for general protobuf usage. (Sadly the direct buffers don't seem to show in heapdumps.) Typically the error is a kernel oom, however, I'm able to consistently reproduce the out of memory error by restricting the direct buffers max memory.

(Image from YourKit in a build that failed and produced the stacktrace below.)

Peeking at the [code](https://github.com/bazelbuild/bazel/blob/7f749e927f61a84ef3a8f361ce601db9d2b81fc4/src/main/java/com/google/devtools/build/lib/runtime/BuildEventStreamer.java#L98-L101) there are many append only sets that in our larger builds begin to cause pressure.

Stacktrace:
```
java.lang.OutOfMemoryError: Cannot reserve 2097152 bytes of direct buffer memory (allocated: 2146624451, limit: 2147483648)
at java.base/java.nio.Bits.reserveMemory(Unknown Source)
at java.base/java.nio.DirectByteBuffer.(Unknown Source)
at java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:701)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:676)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:215)
at io.netty.buffer.PoolArena.tcacheAllocateSmall(PoolArena.java:180)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:137)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:129)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:396)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:124)
at io.grpc.netty.NettyWritableBufferAllocator.allocate(NettyWritableBufferAllocator.java:51)
at io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:226)
at io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:168)
at io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:141)
at io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:65)
at io.grpc.internal.DelayedStream.writeMessage(DelayedStream.java:278)
at io.grpc.internal.ForwardingClientStream.writeMessage(ForwardingClientStream.java:37)
at io.grpc.internal.RetriableStream.sendMessage(RetriableStream.java:545)
at io.grpc.internal.ClientCallImpl.sendMessageInternal(ClientCallImpl.java:520)
at io.grpc.internal.ClientCallImpl.sendMessage(ClientCallImpl.java:506)
at io.grpc.ForwardingClientCall.sendMessage(ForwardingClientCall.java:37)
at io.grpc.stub.ClientCalls$CallToStreamObserverAdapter.onNext(ClientCalls.java:374)
at com.google.devtools.build.lib.buildeventservice.client.BuildEventServiceGrpcClient$BESGrpcStreamContext.sendOverStream(BuildEventServiceGrpcClient.java:166)
at com.google.devtools.build.lib.buildeventservice.BuildEventServiceUploader.publishBuildEvents(BuildEventServiceUploader.java:453)
at com.google.devtools.build.lib.buildeventservice.BuildEventServiceUploader.run(BuildEventServiceUploader.java:311)
at java.base/java.lang.Thread.run(Unknown Source)
```

### Which category does this issue belong to?

Core, Performance

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

The easiest way to reproduce is to restrict the direct buffer pool using `--host_jvm_args=-XX:MaxDirectMemorySize=x`. I was able to get an error on the bazel project itself using the following: `bazelisk --host_jvm_args=-XX:MaxDirectMemorySize=128K build --build_event_json_file=/tmp/bes-out-new.json //...`. This exacerbates the memory contraints. The error is different from above, but ~similar. Toying around with the `-XX:MaxDirectMemorySize` in a build that uses `--bes_backend` would yield a similar error.

### Which operating system are you running Bazel on?

Ubuntu

### What is the output of `bazel info release`?

release 7.0.1 (tho we've had this issue for many past versions as well)

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

_No response_

### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_