apache / apache/hudi

Support data skipping based on record index for flink reader

Open
#17,519 0 comments 0 reactions 1 assignee Claimed by @cshuo View on GitHub
engine:flink type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**
Support data skipping based on record index for flink reader.

**Why this feature is needed:**

Flink reader currently support the following data skipping optimization:
1) partition prune based on partition stats
2) bucket pruning for bucket index
3) file slice pruning based on column stats.

To further improve query performance, we can also support file slice pruning based on record level index.

### User Experience

**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples

### Hudi RFC Requirements

**RFC PR link:** (if applicable)

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.