apache / apache/datafusion-ballista
Enable benchmark data validation for distributed execution
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The TPC-H benchmark suite already has a feature for verifying that results are correct when executing in-memory with DataFusion. It would be good to extend this support to distributed execution with Ballista.
**Describe the solution you'd like**
I would like an option to run the benchmark in data validation mode when executing against a Ballista cluster.
**Describe alternatives you've considered**
None
**Additional context**
None
Contributor guide
Research direction
Start by locating the existing TPC-H benchmark data-validation path for in-memory DataFusion execution, then trace how benchmark options are passed to a Ballista cluster. Add an option for validation during distributed execution and verify that the benchmark reports incorrect results consistently in both execution modes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100