Partial updates do not support incremental query
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
The incremental query does not return values for all columns in the table when we make a partial update; instead, it returns only the fields that changed as part of the partial update.
**What you expected:**
The incremental query should return all fields in the table. Due to this issue, we end up joining the table again to get the complete row, which is a performance hit.
https://apache-hudi.slack.com/archives/C4D716NPQ/p1770644265230019
### Environment
Table type
MOR
**Hudi version:**
Hudi 1.1
**Query engine:** (Spark/Flink/Trino etc)
Spark
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the partial-update scenario described for a MOR table using Hudi 1.1 with Spark, then inspect the incremental query result. Done means the incremental query returns all table columns after a partial update, without requiring a join to recover unchanged fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100