Handle run-away netty buffer thread-local caches

gRPC uses ByteBufs from many non-Netty threads for message serialization. When the application has many threads, the per-thread caches can get out of hand.

Start of relevant discussion: https://gitter.im/grpc/grpc?at=5ac698877c3a01610db0f27b
User solution: https://gitter.im/grpc/grpc?at=5ac796016d7e07082bc71e0a
It references: netty/netty#5930

#1758 is related, as it seems allocating the per-thread cache has a noticeable cost. I thought there may be another related issue, but couldn't find it; maybe I'm thinking of that Netty one.

@jpitz had success just outright disabling caching for non-Netty threads. I'm slightly surprised that didn't cause a CPU increase, as that would seem to mean our direct buffers are no longer cached. If our benchmarks show similar behavior, then SGTM.

However, I also wonder if we should swap to heap buffers for message serialization and framing. Especially for small messages. That might hide most of the cost of removing the cache, since for larger messages the cost of direct buffer allocation is likely dwarfed by any copying.

CC @carl-mastrangelo

Start with the linked discussion, the proposed user solution, Netty issue #5930, and related issue #1758. Compare disabling caching for non-Netty threads with using heap buffers for serialization and framing, especially for small messages. Done should be demonstrated by benchmarks showing the selected approach does not increase CPU usage while preventing runaway per-thread caches.

Scritto dal modello di indicizzazione a partire dal testo della issue.