apache / apache/hudi

[FEATURE] Complete-record output for incremental queries on MoR tables using partial-update / custom-merge payloads

Open
#19,055 2 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**
Make incremental queries (`hoodie.datasource.query.type=incremental`, `incremental.format=latest_state`) on Merge-on-Read (MoR) tables return **complete records** even when the table is written with a **partial-update payload / custom merger** — i.e. when each update writes only `{recordKey, preCombineField, changedColumns}` to the delta log rather than the full record image.

Today, snapshot queries on such tables return fully-merged rows, but incremental queries can return **partial rows** (only the columns present in the in-window log records are populated; all other columns come back `null`). This feature makes the incremental read reconstruct the full record for affected file groups so downstream/derived datasets receive the complete record — the same way Copy-on-Write (CoW) already does.

**Why this feature is needed:**
For partial-update workloads, the upstream intentionally emits only the changed columns per update. On a CoW source this is invisible to consumers, because every commit rewrites the **full** record into the in-window base file, so an incremental read returns the complete row. On a MoR source the same incremental read does not, which makes a CoW → MoR migration **non-transparent** for any derived dataset that consumes the source incrementally (e.g. via Hudi Streamer `HoodieIncrSource`) and assumes the full record arrives from upstream. Downstream merge/upsert logic then overwrites good columns with `null`.

Root cause (read path): the MoR incremental relation builds the queried file slices from **in-window files only**, then merges and applies a post-merge `_hoodie_commit_time` filter:

This "in-window files only" optimization is correct under the implicit assumption that every log record is a complete record image — true for `OverwriteWithLatestAvroPayload` /`DefaultHoodieRecordPayload`, but not for partial-update / custom merge payloads. The feature lifts that assumption for the affected payload types. (Applies to both `MergeOnReadIncrementalRelationV1`for table version 6 and `MergeOnReadIncrementalRelationV2` for table version 8.)

### User Experience

**How users will use this feature:** _Work In Progress_
- Configuration changes needed: _Work In Progress_
- API changes
- No public/user-facing API signature changes. The change is internal to `MergeOnReadIncrementalRelation.collectFileSplits` (file-slice construction) plus one new advanced read config. `HoodieIncrSource` / Hudi Streamer require no code change; the option is passed through as a datasource read option.
- Usage examples: _Work In Progress_

### Hudi RFC Requirements

**RFC PR link:** _Work In Progress_

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? **No** (adds one advanced read config; no API signature changes; behavior can be auto-derived from existing table merge-mode config).
- Does this change storage format? **No** (read-path only; no changes to base/log file layout, timeline, or table properties on disk).
- Justification:

Contributor guide

No contributing guide indexed for this repository

Research direction

Read MergeOnReadIncrementalRelationV1 and MergeOnReadIncrementalRelationV2, focusing on collectFileSplits and the post-merge _hoodie_commit_time filter. Compare the existing CoW behavior and trace how partial-update or custom-merge payloads are identified. Done means latest_state incremental reads on MoR return complete records for both table versions without public API changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.