apache / apache/datafusion-ballista
Improve shuffle (column) statistics
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently, shuffle collects basic statistics like the number of rows and bytes. This data is used downstream for some optimisation. However, I understand those statistics aren’t sufficient for join estimations, leading to incorrect decisions in some cases. No column based statistics such as null count or distinct count is collected during exchange.
**Describe the solution you'd like**
investigate what would be appropriate approach to collect null and distinct count for columns used in repartition expressions. Collecting null count may not be big problem but distinct count might and approximation may be used for it
**Describe alternatives you've considered**
**Additional context**
- collected statistic should be propagated to planners
Contributor guide
Assessment
This issue has not been assessed yet.