apache / apache/hudi

Explore colstats support for shredded variants

Open
#17,988 0 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**

As the title described, shredded variants actually store sub-columns physically, and these should have column statistics.

For reference, complex/nested field vectors are already supported via this PR:
https://github.com/apache/hudi/pull/17694

**Why this task is needed:**

Improve query performance, allowing files to be pruned during query runtime.

### Task Type

Performance optimization

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing PR 17694, which is cited as the reference for complex and nested field vector statistics. Determine how shredded variants store sub-columns and define the colstats changes needed so those columns support runtime file pruning; the task is done when shredded sub-columns have usable column statistics for query pruning.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.