apache / apache/hudi

Partial updates do not support incremental query

Open
#18,943 3 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
The incremental query does not return values for all columns in the table when we make a partial update; instead, it returns only the fields that changed as part of the partial update.

**What you expected:**
The incremental query should return all fields in the table. Due to this issue, we end up joining the table again to get the complete row, which is a performance hit.

https://apache-hudi.slack.com/archives/C4D716NPQ/p1770644265230019

### Environment
Table type
MOR
**Hudi version:**
Hudi 1.1
**Query engine:** (Spark/Flink/Trino etc)
Spark

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the partial-update scenario described for a MOR table using Hudi 1.1 with Spark, then inspect the incremental query result. Done means the incremental query returns all table columns after a partial update, without requiring a join to recover unchanged fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.