apache / apache/datafusion-ballista

Enable benchmark data validation for distributed execution

Open
#454 2 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The TPC-H benchmark suite already has a feature for verifying that results are correct when executing in-memory with DataFusion. It would be good to extend this support to distributed execution with Ballista.

**Describe the solution you'd like**
I would like an option to run the benchmark in data validation mode when executing against a Ballista cluster.

**Describe alternatives you've considered**
None

**Additional context**
None

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing TPC-H benchmark data-validation path for in-memory DataFusion execution, then trace how benchmark options are passed to a Ballista cluster. Add an option for validation during distributed execution and verify that the benchmark reports incorrect results consistently in both execution modes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.