opensearch-project / opensearch-project/data-prepper
[question] Confusion on performance test results
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Hi, I saw in the latest test results that a throughput of 19,684 was achieved with a batch size of 200 in the DataPrepper test. However, further down on that page in the DataPrepper pipeline configuration that was given, the batch size is listed as 5000. Can someone help me distinguish between these two values and help me understand where the 5000 has come from?
This is part of a larger question I am researching, which is what my batch size needs to be set to in order to achieve a certain throughput. If anyone can help me understand these results or this question in general, that would be very helpful as I haven't found anything else related to this in the documentation. Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read docs/latest_performance_test_results.md at the linked commit, comparing the reported batch size of 200 with the DataPrepper pipeline configuration showing 5000. Clarify what each value represents and document how readers should use the results when choosing a batch size for a target throughput.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation, performance
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100