awslabs / awslabs/amazon-msk-data-generator
Largest Supported Messages/Second Inputted and Optimal Worker Connector Configurations
- Dominant language
- Java
- Stars
- 18
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
I am currently trying out the MSK Data Generator as an alternative to Voluble thanks to its direct connection to AWS MSK. However, I'm noticing that the data generated/second seems to be slower than what we were getting locally on our Voluble tests. We were wondering what the maximum supported messages/second is estimated to be, and if this Data Generator is optimized similarly to Voluble.
For reference, on a cluster with the broker type of kafka.m5.2xlarge and 2 brokers (over 2 AZs), we are getting the following as our messages generated:
Autoscaled:
With 6 workers and 1 MCU per, ~1375 messages per second
With 3 workers and 2 MCU per, ~1500 messages per second
With 7 workers and 1 MCU per, ~1464 messages per second
Provisioned:
With 6 workers and 1 MCU per, ~1700 messages per second.
Obviously, this seems quite underutilized and we were wondering if there was something wrong with the connector configuration or general optimization. We are aiming to try to generate 1 million messages per second, and are unsure if that is achievable even with a kafka.m5.24xlarge broker size.
Additionally, the connectors seem to fail when we try to modify the connector capacity, MCU count/worker, and worker count, we also notice that the connector fails. There's no documentation available regarding this optimization, and I'd love to contribute my findings to help, but I was wondering if there is any known optimal value set. I've noticed that maxing the connector capacity to that of the broker size results in the failure of the connector (ie m5.2xlarge supports up to 32 GB and 8 vCPUs but setting a configuration such as 4 workers and 2 MCUs per results in the connector failing, while 3 workers and 2 MCUs works fine where each MCU needs one vCPU and 4 GB of RAM). I've also encountered issues with a m5.24xlarge creating any connectors at all, where the connector eventually transitions into a failed state in both provisioned and autoscaled capacity types.
Would love to know if there is something wrong with our optimization or our connector usage and/or if there has been any testing done with a similar set up, as we have followed the deployment steps found within this repository [here](https://github.com/awslabs/amazon-msk-data-generator/blob/2712203a1f697fa8fb682de517cdaba98fe26cdd/docs/msk-connect-deploy.md).
Contributor guide
Research direction
The issue points to docs/msk-connect-deploy.md; start by reviewing its deployment steps alongside the reported autoscaled and provisioned worker/MCU configurations. Reproduce or compare the listed throughput and connector failures, then document confirmed limits, working configurations, and failure conditions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kafka
- Domain
- backend, cloud, data-engineering
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100