redpanda-data / redpanda-data/openmessaging-benchmark

document in driver-redpanda producer workload tunings for high volume producer configs.

Open
#76 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
40
Forks
32
PR merge metrics
No merged PRs in 30d

Description

https://redpandadata.slack.com/archives/C01ND4SVB6Z/p1694729911166379

Need to document some additional OMB workload configuration detail for high volume testing.

There's a few items in this thread talking about how to get the producer to keep up with the expected rates

  1. Possible quirks with key distributor not keeping up with the expected rate. Random Nano seems to act weird in high rate setups across many producers and partition spreads. NoKey and Round Robin seem to keep up. this could be related to next issue
  2. for high volume produce rates (>1million/s) across many producers (tens) going to many partitions (thousands), java client may also need to have buffer.memory significantly increased to handle the amount of data being generated in the batch.

In example test, @travisdowns calculated that for 1.8m messages/sec on 10 partitions with thousands of partitions each coming from ~100 producers, the buffer size needed was likely 3-4x larger than what we were setting in the test (around 32-33MB).

2300 partitions per topic * 32000 batch size = 73.6 MB

according to java client docs, when doing larger batch sizes

A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.

so we may not have been able to fill the batch due to buffer limits in the original tests.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Slack thread and inspect the driver-redpanda producer workload configurations used for high-volume tests. Document the key-distributor behavior and the Java client's buffer.memory guidance for large batches, including the reported partition and producer scale. Done means these tuning details are recorded where OMB workload configuration documentation belongs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
documentation, performance
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.