apache / apache/hudi

Introduce materialization into HoodieBackedTableMetadata

Open
#15,580 1 comment 0 reactions 0 assignees View on GitHub
area:metadata-table from-jira priority:critical type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

*Problem Statement*

Currently, MT performance is hardly predictable due to variety of factors such as, for ex, 
whether the MT is compacted: if table is NOT compacted, when loading "files" partition for ex, we will load all of the delta-log files materializing them in-memory,  meaning that all subsequent requests will be served from memory. However, when table IS compacted, we will only prematerialize the updated records but not the records sitting in the base file, which would require us to go fetch from base HFile every time (even though there's block-level caching implemented inside HFile reader).

More generally, `HoodieBackedTableMetadata` being the primary facade and interface for MT, currently doesn't have a well thought-through architecture and APIs, instead it serves simply as an aggregation layer for the lower-level components (LogRecordScanner, FileReader, etc).

This is problematic, since MT is a core component performance of which has direct implication on the query planning and beyond. As such, it has to have:
# {*}Predictable performance{*}: how state of MT affects performance should be easy to comprehend and reason about (for ex, {_}it's expected that performance could be decreasing, with increase in scale or if the table is not compacted for a long time; however it's totally unexpected that performance could become worse than it was after compaction{_})
# {*}Have clear configuration levers{*}: behavior, performance of the MT should have crystal clear configuration levers – whether records are materialized in-memory or loaded dynamically, 

 

*Solution*

To address aforementioned problems, we propose to implement HoodieBackedTableMetadataV2 providing
* {*}Materialization{*}: it should allow MT to be read in either of 2 ways
** _Eagerly:_ when whole MT is loaded in-memory before accessing
** _Lazily:_ when MT is queried on an ad-hoc basis, however caching the results of the previous queries for subsequent use
*  

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5236
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1292

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading HoodieBackedTableMetadata and its lower-level LogRecordScanner and FileReader components, then review the linked HUDI-5236 proposal. Define how HoodieBackedTableMetadataV2 will support eager and lazy materialization, including caching for lazy reads. Done means both access modes and their configuration levers are implemented and covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.