apache / apache/datafusion

Adjust `Statistics::total_byte_size` in `Filter` with a projection

Open
#13,224 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

@Dandandan pointed out in https://github.com/apache/datafusion/pull/13187/files#r1824330274 that when applying a projection to a filter the resulting `StatisticsCalculation` is not properly updated

> I think the global stats (`total_byte_size`) are not correct either, doesn't take into account the reduced number of columns. It should do something similar as `stats_projection` for `ProjectionExec`

However, I did not want to try and add that in the bugfix PR https://github.com/apache/datafusion/pull/13187 because:
- The `total_byte_size` calculation in filter also needs to take estimated selectivity into account
- The calculation of `total_byte_size` in `stats_projection` is also somewhat suspect as it only accounts for "fixed sized" rows but still claims the size is known precisely

### Describe the solution you'd like

Account for the projection in the filter statistics calculation somehow

### Describe alternatives you've considered

https://github.com/apache/datafusion/blob/ac79ef3442e65f6197c7234da9fad964895b9101/datafusion/physical-plan/src/projection.rs#L261-L260

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the linked stats_projection implementation in datafusion/physical-plan/src/projection.rs and compare it with the filter statistics calculation discussed in PR 13187. Determine how projection and estimated selectivity should affect total_byte_size, then verify that the resulting filter statistics account for the reduced columns without claiming unsupported precision.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.