MERGE INTO with a partial UPDATE SET writes a misaligned record on MOR with a global bloom or simple index
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
On a Merge-on-Read table using a global bloom or global simple index, a `MERGE INTO` with a partial `UPDATE SET` fails while serializing the merged payload:
```
org.apache.avro.UnresolvedUnionException: Not in union ["null","long"]: 15.0 (field=ts)
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:247)
at org.apache.hudi.common.avro.HoodieAvroUtils.avroToBytes(HoodieAvroUtils.java:182)
at org.apache.hudi.common.model.BaseAvroPayload.getRecordBytes(BaseAvroPayload.java:111)
at org.apache.hudi.common.model.DefaultHoodieRecordPayload.write(DefaultHoodieRecordPayload.java:214)
at com.esotericsoftware.kryo...KryoSerializableSerializer.write
at org.apache.hudi.common.model.HoodieRecord.write(HoodieRecord.java:406)
```
`amount`'s value is landing in `ts`'s slot. The record is malformed during index tagging and the failure surfaces the moment Spark serializes it to shuffle the tagged records, before any log block is written.
**To reproduce**
```sql
-- hoodie.index.type = GLOBAL_BLOOM
-- hoodie.bloom.index.update.partition.path = false
CREATE TABLE t (id BIGINT, name STRING, amount DOUBLE, ts BIGINT, dt STRING)
USING hudi PARTITIONED BY (dt)
TBLPROPERTIES (type = 'mor', primaryKey = 'id', preCombineField = 'ts');
INSERT INTO t VALUES (1, 'a', 10.0, 1, '2026-08-11');
MERGE INTO t AS t
USING (SELECT 1L AS id, 15.0 AS amount, 200L AS ts, '2026-08-11' AS dt) AS s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET t.amount = s.amount, t.ts = s.ts;
```
Expected: the row updates in place, `name` and `dt` keeping their existing values, one row, still in `dt=2026-08-11`.
Not assigning the record key is the ordinary shape for a partial update: `hoodie.spark.sql.merge.into.partial.updates` defaults to `true`, and `isPartialUpdateActionForMOR` deliberately waives the record-key assignment requirement.
**Scope**
| Configuration | Result |
| --- | --- |
| MOR, `GLOBAL_BLOOM` or `GLOBAL_SIMPLE` | fails |
| MOR, `BLOOM` or `SIMPLE` (non-global) | works |
| MOR, `RECORD_INDEX` or `GLOBAL_RECORD_LEVEL_INDEX` | works |
| Copy-on-Write, any of the above | works |
| same statement with the record key assigned | works |
Those two index types are the only ones that both pre-merge at tagging time and then inspect the merged record: `mayContainDuplicateLookup` is `tableType == MERGE_ON_READ` for global bloom and simple, while the record-index implementations pass `false` and short-circuit, and CoW defers the merge to the file rewrite. Assigning the record key works because it makes `areAllFieldsUpdated` true, switching partial updates off.
Verified present on released 1.1.1, on 1.2.0-SNAPSHOT, and on a bundle built from the 1.2.1 release-staging branch.
**Cause**
Under partial updates the merge produces a record carrying only the assigned columns, which is intended: `HoodieAppendHandle` takes the partial schema as its writer schema, and `BaseWriteHelper` does the same for dedup. But `HoodieIndexUtils.mergeIncomingWithExistingRecordWithExpressionPayload` wraps that record against the full write schema, and `HoodieAvroIndexedRecord#prependMetaFields` infers the meta-field count as `targetSchema.size() - record.size()`. For a two-field record against a ten-field target that is 8, so `JoinedGenericRecord` treats eight slots as meta and the data lands at indices 8 and 9 instead of 7 and 8.
**Also on the same path**
`inferPartitionPath` derives the partition from that same merged record, and `KeyGenUtils#getPartitionPath` substitutes the default partition for a field it cannot find, so the payload already carries `dt=__HIVE_DEFAULT_PARTITION__` while both the incoming and existing records are in the real partition. It is harmless today only because the write fails first: fixing the schema alone would turn a loud failure into a silently mis-partitioned row.
**Note**
This is only reachable once #19708 is fixed. Before that, the same statement fails earlier with `HoodieKeyException: recordKey value: "null" for field: "id"` from `SqlKeyGenerator.getPartitionPath`.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.