Abstract out generic FileIndex
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
FileIndex has to be providing its full-suite of capabilities in a generic way:
* Partition Pruning
* Column Stats Pruning
* Caching
* etc
To support partition-pruning as well as col-stats pruning in an engine-agnostic way, we'd have to implement our own Expression hierarchy supporting
* Conversion from engine-specific one to Hudi's hierarchy
* Being able to execute expressions
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5135
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-4141
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing Hudi's existing FileIndex implementation and the linked JIRA work, HUDI-5135 and HUDI-4141. The design needs to cover engine-agnostic partition pruning, column-statistics pruning, caching, and an expression hierarchy that converts engine-specific expressions and executes them; done means these capabilities are abstracted generically.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100