[FEATURE] Complete-record output for incremental queries on MoR tables using partial-update / custom-merge payloads
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
**What the feature achieves:**
Make incremental queries (`hoodie.datasource.query.type=incremental`, `incremental.format=latest_state`) on Merge-on-Read (MoR) tables return **complete records** even when the table is written with a **partial-update payload / custom merger** — i.e. when each update writes only `{recordKey, preCombineField, changedColumns}` to the delta log rather than the full record image.
Today, snapshot queries on such tables return fully-merged rows, but incremental queries can return **partial rows** (only the columns present in the in-window log records are populated; all other columns come back `null`). This feature makes the incremental read reconstruct the full record for affected file groups so downstream/derived datasets receive the complete record — the same way Copy-on-Write (CoW) already does.
**Why this feature is needed:**
For partial-update workloads, the upstream intentionally emits only the changed columns per update. On a CoW source this is invisible to consumers, because every commit rewrites the **full** record into the in-window base file, so an incremental read returns the complete row. On a MoR source the same incremental read does not, which makes a CoW → MoR migration **non-transparent** for any derived dataset that consumes the source incrementally (e.g. via Hudi Streamer `HoodieIncrSource`) and assumes the full record arrives from upstream. Downstream merge/upsert logic then overwrites good columns with `null`.
Root cause (read path): the MoR incremental relation builds the queried file slices from **in-window files only**, then merges and applies a post-merge `_hoodie_commit_time` filter:
This "in-window files only" optimization is correct under the implicit assumption that every log record is a complete record image — true for `OverwriteWithLatestAvroPayload` /`DefaultHoodieRecordPayload`, but not for partial-update / custom merge payloads. The feature lifts that assumption for the affected payload types. (Applies to both `MergeOnReadIncrementalRelationV1`for table version 6 and `MergeOnReadIncrementalRelationV2` for table version 8.)
### User Experience
**How users will use this feature:** _Work In Progress_
- Configuration changes needed: _Work In Progress_
- API changes
- No public/user-facing API signature changes. The change is internal to `MergeOnReadIncrementalRelation.collectFileSplits` (file-slice construction) plus one new advanced read config. `HoodieIncrSource` / Hudi Streamer require no code change; the option is passed through as a datasource read option.
- Usage examples: _Work In Progress_
### Hudi RFC Requirements
**RFC PR link:** _Work In Progress_
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? **No** (adds one advanced read config; no API signature changes; behavior can be auto-derived from existing table merge-mode config).
- Does this change storage format? **No** (read-path only; no changes to base/log file layout, timeline, or table properties on disk).
- Justification:
Contributor guide
No contributing guide indexed for this repository
Research direction
Read MergeOnReadIncrementalRelationV1 and MergeOnReadIncrementalRelationV2, focusing on collectFileSplits and the post-merge _hoodie_commit_time filter. Compare the existing CoW behavior and trace how partial-update or custom-merge payloads are identified. Done means latest_state incremental reads on MoR return complete records for both table versions without public API changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100