Unlimited file-based data generation
- Dominant language
- Python
- Stars
- 2k
- Forks
- 347
- Avg merge
- 6d 17h
- Merged PRs (30d)
- 5
Description
The using the default parameter source to generate data based on files, the content is divided up between processes and the benchmark terminates once all data has been ingested. This makes it difficult to create duration-based benchmarks as they may finish prematurely for very powerful nodes/clusters.
It would be very useful if the standard, file-based parameter source could be configured to generate an infinite stream of data by e.g. looping the provided files and just updating the timestamp to correspond to `now`.
Contributor guide
Research direction
Start by locating Rally’s default file-based parameter source and inspect how it partitions file content between processes and decides that ingestion is complete. The feature is done when the source can loop the files indefinitely, refresh timestamps relative to now, and support duration-based benchmarks without premature termination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100