Derive per-query aggregation-heaviness + NDV metadata
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Whether dictionary-encoding a column helps depends not just on its cardinality but on how heavily it's aggregated in the query. There's no existing signal that combines both.
### Describe the solution you'd like
Derive metadata that combines NDV with an estimate of aggregation "heaviness" for a given query plan. Heavy aggregation + low NDV = more room for error/benefit if we decide to cast to dictionary arrays early; this signal should be usable by the optimizer.
### Describe alternatives you've considered
_No response_
### Additional context
sub-issue of : #22891
part of epic : https://github.com/apache/datafusion/issues/24111
Contributor guide
Research direction
Start by reading the parent issue #22891 and the epic #24111 to understand the intended optimizer context, then inspect the query-plan metadata areas they point toward. Done means deriving a per-query signal that combines NDV with estimated aggregation heaviness and making that signal usable by the optimizer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100