benchmarking producer client issue with high volume (1.5 million/sec) low size (100 byte) messages

Recently in attempting to generate traffic to provide 1.5 GB/sec to an appropriately sized cluster, using 100 byte message sizes (customer is moving from SQS) we have encountered difficulties with the OMB producers. Messages per second seems to fall apart somewhere around 1 to 1.2 million messages per second. The same cluster can handle 1.5GB w/ 1024 message sizes with excellent performance. 1.5GB w 100 byte messages (same batch size, etc) results in producers erroring out and aborting.

@travisdowns has additional details around the nature of these failures he can add to this issue.

Additional experiments using other client technologies have been in progress by @larsenpanda .

Ultimately we need to update our client so that these types of workloads are successful out of the box and define limits for the client in documentation so customers know what the upper boundaries of the producers and consumers are so they do not infer poor performance by Redpanda.

Start by reproducing the OMB producer workload described: compare 100-byte messages at roughly 1–1.5 million messages per second with the successful 1,024-byte workload, and collect the producer errors and abort conditions. Done means the client handles the target workload out of the box and the producer and consumer limits are documented.