apache / apache/datafusion

Derive per-query aggregation-heaviness + NDV metadata

Open
#24,116 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Whether dictionary-encoding a column helps depends not just on its cardinality but on how heavily it's aggregated in the query. There's no existing signal that combines both.

### Describe the solution you'd like

Derive metadata that combines NDV with an estimate of aggregation "heaviness" for a given query plan. Heavy aggregation + low NDV = more room for error/benefit if we decide to cast to dictionary arrays early; this signal should be usable by the optimizer.

### Describe alternatives you've considered

_No response_

### Additional context

sub-issue of : #22891
part of epic : https://github.com/apache/datafusion/issues/24111

Contributor guide

Open the contributing guide

Research direction

Start by reading the parent issue #22891 and the epic #24111 to understand the intended optimizer context, then inspect the query-plan metadata areas they point toward. Done means deriving a per-query signal that combines NDV with estimated aggregation heaviness and making that signal usable by the optimizer.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.