apache / apache/datafusion

Enabling `datafusion.explain.show_statistics` by default

Open
#7,504 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Follow up based on https://github.com/apache/arrow-datafusion/pull/7459#pullrequestreview-1616135946

Displaying statistics in executions plans as a default behaviour is nice to have feature. The only challenge here is to enable/implement it without significantly growing the burden of tests maintenance, as some of them, in different parts of codebase, rely on text representation on execution plan, and it doesn't seem convenient to fix random tests across multiple packages in case of e.g. statistics calculation related changes.

### Describe the solution you'd like

One possible solution might be to explicitly disabling `show_statistics` setting everywhere in tests, except for cases requiring statistics in output.

Another one could be to have a common normalizer (like we already have in [sqllogictests](https://github.com/apache/arrow-datafusion/blob/4d4451294629940d340160cdd06be273139728b4/datafusion/sqllogictest/src/engines/datafusion_engine/normalize.rs#L32) and in [core/tests/sql](https://github.com/apache/arrow-datafusion/blob/4d4451294629940d340160cdd06be273139728b4/datafusion/core/tests/sql/mod.rs#L806) for datafiles path replacement) which could be accessible across all tests and would allow to replace statistics with empty or remove it from plan text. In this case the `datafusion.explain.show_statistics` setting is not needed anymore.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the existing normalizers in sqllogictest/src/engines/datafusion_engine/normalize.rs and core/tests/sql/mod.rs, then review tests that assert execution-plan text. Decide how show_statistics can be enabled by default while keeping plan assertions stable, and verify that statistics-specific output remains covered without widespread brittle updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.