elastic / elastic/rally

Unlimited file-based data generation

Open
#294 4 comments 1 reaction 0 assignees View on GitHub
:Load Driver :Track Management enhancement
Dominant language
Python
Stars
2k
Forks
347
Avg merge
6d 17h
Merged PRs (30d)
5

Description

The using the default parameter source to generate data based on files, the content is divided up between processes and the benchmark terminates once all data has been ingested. This makes it difficult to create duration-based benchmarks as they may finish prematurely for very powerful nodes/clusters.

It would be very useful if the standard, file-based parameter source could be configured to generate an infinite stream of data by e.g. looping the provided files and just updating the timestamp to correspond to `now`.

Contributor guide

Open the contributing guide

Research direction

Start by locating Rally’s default file-based parameter source and inspect how it partitions file content between processes and decides that ingestion is complete. The feature is done when the source can loop the files indefinitely, refresh timestamps relative to now, and support duration-based benchmarks without premature termination.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.