Adjust `Statistics::total_byte_size` in `Filter` with a projection
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
@Dandandan pointed out in https://github.com/apache/datafusion/pull/13187/files#r1824330274 that when applying a projection to a filter the resulting `StatisticsCalculation` is not properly updated
> I think the global stats (`total_byte_size`) are not correct either, doesn't take into account the reduced number of columns. It should do something similar as `stats_projection` for `ProjectionExec`
However, I did not want to try and add that in the bugfix PR https://github.com/apache/datafusion/pull/13187 because:
- The `total_byte_size` calculation in filter also needs to take estimated selectivity into account
- The calculation of `total_byte_size` in `stats_projection` is also somewhat suspect as it only accounts for "fixed sized" rows but still claims the size is known precisely
### Describe the solution you'd like
Account for the projection in the filter statistics calculation somehow
### Describe alternatives you've considered
https://github.com/apache/datafusion/blob/ac79ef3442e65f6197c7234da9fad964895b9101/datafusion/physical-plan/src/projection.rs#L261-L260
### Additional context
_No response_
Contributor guide
Research direction
Start with the linked stats_projection implementation in datafusion/physical-plan/src/projection.rs and compare it with the filter statistics calculation discussed in PR 13187. Determine how projection and estimated selectivity should affect total_byte_size, then verify that the resulting filter statistics account for the reduced columns without claiming unsupported precision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100