apache / apache/hudi

Chain multiple indices support in HoodieFileIndex

Open
#17,267 1 comment 0 reactions 1 assignee Assigned to @danny0405 View on GitHub
area:reader from-jira priority:critical type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

In HoodieFileIndex, we have 5 indices support: 

 
* RecordLevelIndexSupport
* BucketIndexSupport
* SecondaryIndexSupport
* FunctionalIndexSupport
* BloomFiltersIndexSupport
* ColumnStatsIndexSupport

 
But the logic would return early when the first index support applied and got pruned files.
A better way is we can chain the multiple indices support all-together: one index support can still be applied based on the pruned files returned by another index support.

Key points in contribution:
1. think through the shared contract/param among these indices support;
2. add a orthoganal test for random index supoprt composition, valid the result set and the indices support are applied effectively.
 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8164
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9101
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.