apache / apache/datafusion

Join per-column stats not scaled with estimated row count

Open
#22,761 1 comment 0 reactions 1 assignee Claimed by @neilconway View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

For inner/left/right/full joins, estimate_join_cardinality concatenates both inputs' column statistics verbatim and never adjusts them to the estimated output cardinality:

- Count-like fields (null_count, distinct_count, byte_size, sum_value) aren't scaled to the join's fan-out/selectivity, so they can be inconsistent with the output num_rows.
- Outer joins don't NULL-pad the non-preserved side, even though the side-to-pad is already known (utils.rs:240-242).
- Equi-join key columns aren't intersected (min/max) or NDV-bounded (min(left_ndv, right_ndv)).

Related to #22743 but the general join case might be a bit more involved.

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.