apache / apache/datafusion

Consolidate benchmarking options/ENV settings

Open
#21,936 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

- part of https://github.com/apache/datafusion/issues/21706

as mentioned several places on
- https://github.com/apache/datafusion/pull/21707

There are several command line flags / environment variables that are duplicated in the benchmark runner (and @Omega359 just followed the same pattern with the new runner)

It would be great to consolidate all the options, and for any that have existing environment settings from SessionConfig, instead use session config

It feels like we maybe could just reuse https://docs.rs/datafusion/latest/datafusion/prelude/struct.SessionConfig.html#method.from_env rather than implementing special cases for many of these options

_Originally posted by @alamb in https://github.com/apache/datafusion/pull/21707#discussion_r3121443426_

Contributor guide

Open the contributing guide

Research direction

Start with the benchmark runner and the option handling discussed in issues/21706 and pull/21707, then read SessionConfig::from_env in the DataFusion documentation. Compare the duplicated command-line flags and environment settings across runners and identify which can use SessionConfig. Done means the options are consolidated without separate special cases for settings already supported by SessionConfig.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.