MERGE INTO with a partial UPDATE SET fails on MOR with a global bloom or simple index
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
On a Merge-on-Read table using a global bloom or global simple index, a `MERGE INTO` whose `UPDATE SET` does not assign the record key fails inside the writer:
```
org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time ...
at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:84)
at org.apache.hudi.table.action.deltacommit.SparkUpsertDeltaCommitActionExecutor.execute(SparkUpsertDeltaCommitActionExecutor.java:45)
Cause: org.apache.hudi.exception.HoodieKeyException:
recordKey value: "null" for field: "id" cannot be null or empty
at org.apache.hudi.keygen.KeyGenUtils.getRecordKey(KeyGenUtils.java:270)
at org.apache.hudi.keygen.SimpleAvroKeyGenerator.getRecordKey(SimpleAvroKeyGenerator.java:50)
at org.apache.spark.sql.hudi.command.SqlKeyGenerator.$anonfun$getPartitionPath$1(SqlKeyGenerator.scala:102)
```
Not assigning the record key is the ordinary shape for a partial update, so this makes partial-update `MERGE INTO` unusable on that combination.
**To reproduce**
```sql
-- hoodie.index.type = GLOBAL_BLOOM
-- hoodie.bloom.index.update.partition.path = false
CREATE TABLE t (id BIGINT, name STRING, amount DOUBLE, ts BIGINT, dt STRING)
USING hudi PARTITIONED BY (dt)
TBLPROPERTIES (type = 'mor', primaryKey = 'id', preCombineField = 'ts');
INSERT INTO t VALUES (1, 'a', 10.0, 1, '2026-08-11');
MERGE INTO t AS t
USING (SELECT 1L AS id, 15.0 AS amount, 200L AS ts, '2026-08-11' AS dt) AS s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET t.amount = s.amount, t.ts = s.ts;
```
Expected: the row updates in place, with `name` and `dt` keeping their existing values.
**Scope**
| Configuration | Result |
| --- | --- |
| MOR, `GLOBAL_BLOOM` or `GLOBAL_SIMPLE` | fails |
| MOR, `RECORD_INDEX` or `GLOBAL_RECORD_LEVEL_INDEX` | works |
| Copy-on-Write, any of the above | works |
| Same statement with the record key assigned | works |
The index types split because `mayContainDuplicateLookup` is `tableType == MERGE_ON_READ` for global bloom and global simple, while the record-index implementations pass `false` and never reach the merge stage. Assigning the record key works because it makes `areAllFieldsUpdated` true, which turns partial updates off.
**Cause**
`SqlKeyGenerator#getPartitionPath(GenericRecord)` resolves the partition path by way of `BaseKeyGenerator#getKey`, which is `new HoodieKey(getRecordKey(record), getPartitionPath(record))`. Asking for a partition path therefore also validates the record key. On the path above, `HoodieIndexUtils#inferPartitionPath` asks for the partition path of a merged record materialised against `WRITE_PARTIAL_UPDATE_SCHEMA`, which carries only the columns named in `UPDATE SET`, so the record key is legitimately absent.
**Note**
Fixing this exposes a second, currently unreachable problem on the same path, where the partial-update merged record is serialized against a mismatched schema and raises `UnresolvedUnionException` from `BaseAvroPayload#getRecordBytes`. That has a distinct cause in the payload path and will be filed separately.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.