apache / apache/hudi

MERGE INTO with a partial UPDATE SET fails on MOR with a global bloom or simple index

Open
#19,708 1 comment 0 reactions 1 assignee Claimed by @linliu-code View on GitHub
area:sql priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

On a Merge-on-Read table using a global bloom or global simple index, a `MERGE INTO` whose `UPDATE SET` does not assign the record key fails inside the writer:

```
org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time ...
at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:84)
at org.apache.hudi.table.action.deltacommit.SparkUpsertDeltaCommitActionExecutor.execute(SparkUpsertDeltaCommitActionExecutor.java:45)
Cause: org.apache.hudi.exception.HoodieKeyException:
recordKey value: "null" for field: "id" cannot be null or empty
at org.apache.hudi.keygen.KeyGenUtils.getRecordKey(KeyGenUtils.java:270)
at org.apache.hudi.keygen.SimpleAvroKeyGenerator.getRecordKey(SimpleAvroKeyGenerator.java:50)
at org.apache.spark.sql.hudi.command.SqlKeyGenerator.$anonfun$getPartitionPath$1(SqlKeyGenerator.scala:102)
```

Not assigning the record key is the ordinary shape for a partial update, so this makes partial-update `MERGE INTO` unusable on that combination.

**To reproduce**

```sql
-- hoodie.index.type = GLOBAL_BLOOM
-- hoodie.bloom.index.update.partition.path = false
CREATE TABLE t (id BIGINT, name STRING, amount DOUBLE, ts BIGINT, dt STRING)
USING hudi PARTITIONED BY (dt)
TBLPROPERTIES (type = 'mor', primaryKey = 'id', preCombineField = 'ts');

INSERT INTO t VALUES (1, 'a', 10.0, 1, '2026-08-11');

MERGE INTO t AS t
USING (SELECT 1L AS id, 15.0 AS amount, 200L AS ts, '2026-08-11' AS dt) AS s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET t.amount = s.amount, t.ts = s.ts;
```

Expected: the row updates in place, with `name` and `dt` keeping their existing values.

**Scope**

| Configuration | Result |
| --- | --- |
| MOR, `GLOBAL_BLOOM` or `GLOBAL_SIMPLE` | fails |
| MOR, `RECORD_INDEX` or `GLOBAL_RECORD_LEVEL_INDEX` | works |
| Copy-on-Write, any of the above | works |
| Same statement with the record key assigned | works |

The index types split because `mayContainDuplicateLookup` is `tableType == MERGE_ON_READ` for global bloom and global simple, while the record-index implementations pass `false` and never reach the merge stage. Assigning the record key works because it makes `areAllFieldsUpdated` true, which turns partial updates off.

**Cause**

`SqlKeyGenerator#getPartitionPath(GenericRecord)` resolves the partition path by way of `BaseKeyGenerator#getKey`, which is `new HoodieKey(getRecordKey(record), getPartitionPath(record))`. Asking for a partition path therefore also validates the record key. On the path above, `HoodieIndexUtils#inferPartitionPath` asks for the partition path of a merged record materialised against `WRITE_PARTIAL_UPDATE_SCHEMA`, which carries only the columns named in `UPDATE SET`, so the record key is legitimately absent.

**Note**

Fixing this exposes a second, currently unreachable problem on the same path, where the partial-update merged record is serialized against a mismatched schema and raises `UnresolvedUnionException` from `BaseAvroPayload#getRecordBytes`. That has a distinct cause in the payload path and will be filed separately.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.