apache / apache/datafusion-ballista

Improve shuffle (column) statistics

Open
#1,827 1 comment 0 reactions 1 assignee Claimed by @coderfender View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently, shuffle collects basic statistics like the number of rows and bytes. This data is used downstream for some optimisation. However, I understand those statistics aren’t sufficient for join estimations, leading to incorrect decisions in some cases. No column based statistics such as null count or distinct count is collected during exchange.

**Describe the solution you'd like**

investigate what would be appropriate approach to collect null and distinct count for columns used in repartition expressions. Collecting null count may not be big problem but distinct count might and approximation may be used for it

**Describe alternatives you've considered**

**Additional context**

- collected statistic should be propagated to planners

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.