Incremental read surpports latest_state_with_before format
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The current incremental query for the time range *(t1, t2]* returns the latest record values at *t2.* However, there are situations where we need the record values at *t1* to obtain insert/update information. This is particularly useful in cases of data rollback, as we need to retrieve this delta information(insert/update) from Hudi to roll back external storage (such as Hbase in downstream systems).
Hudi RFC-51 introduces CDC (Change Data Capture) support, but {*}it returns all commit change records between (t1, t2]{*}. This can be somewhat inefficient and wasteful of resources when we only want to know the values at t1 and t2. Furthermore, the returned records in CDC mode are in {*}JSON format with a unified schema(a customized format for CDC scenarios){*}, which differs from the user's original schema. This makes using CDC more difficult compared to snapshot/incremental reads.
An alternative option to obtain both record values at t1 and t2 is to rely on the existing incremental query implementation.
We can include the record value at t1 (before value) when returning the latest record value at t2 in incremental read. For example:
When `hoodie.datasource.query.incremental.format=latest_state`, the returned record is:
{code:java}
column1: string, column2: array{code}
When `hoodie.datasource.query.incremental.format=latest_state_with_before`, the returned record is:
{code:java}
column1: string, column1_before_: string, column2: array, column2_before_: array,{code}
The implementation is simple. Sort both the base file and log file in advance(e.g., compact in MOR), and then perform an ordered merge to combine them.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6409
- Type: Improvement
- Attachment(s):
- 19/Jun/23 07:23;dennysong;incremental_format.svg;https://issues.apache.org/jira/secure/attachment/13059192/incremental_format.svg
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating Hudi's existing incremental query implementation and the handling for the `latest_state` format. Trace how base-file and log-file records are read and merged, then define tests covering `latest_state_with_before` with the documented before-value column names; done means both t1 and t2 values are returned without changing the existing format.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100