apache / apache/hudi

[BUG] Payload-based MOR tables mis-decode variant rows once Spark 4.1 PushVariantIntoScan fires

Open
#19,768 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Split out of the #18923 review (https://github.com/apache/hudi/pull/18923#discussion_r3859930172). On a MOR table whose merger is payload-based (merge mode CUSTOM with a user payload class, strategy `00000000-0000-0000-0000-000000000000`), a Spark 4.1+ query that lets PushVariantIntoScan rewrite a variant column (for example `cast(v as string)`) reads base rows in the projected struct shape but log rows as plain VariantVal, and the two shapes meet in the avro round trip the payload merger performs:

- base + log for the same key: `HoodieAvroRecordMerger.merge` converts the base row to avro against a schema that still types the column as VARIANT, so the projected struct bytes get decoded as a variant.
- log-only key: `PayloadUpdateProcessor.handleNonDeletes` rewrites the record through avro and hands back a VariantVal-shaped row, which `HoodieFileGroupReaderBasedFileFormat.appendPartitionAndProject` then projects as if it were the struct.

Traced from the code, not yet reproduced. It predates #18923: the base-side overlay in `SparkFileFormatInternalRowReaderContext.getFileRecordIterator` has been unconditional since #18674, and the payload skip on the log side (`payloadClasses.isPresent()` in the old `FileGroupRecordBuffer.getProjectedTransformer`, `shouldProjectVariants()` after #18923) only stops the log rows from being rewritten. The same gap was pointed out on #18674 (https://github.com/apache/hudi/pull/18674#discussion_r3243731762) and never picked up.

**To Reproduce**

Expected trigger, unrun. Spark SQL under the spark4.1 or spark4.2 profile (Spark 4.0 has no PushVariantIntoScan):

```sql
create table t (id int, v variant, ts long) using hudi
location '/tmp/variant_payload_repro'
tblproperties (
primaryKey = 'id', type = 'mor', preCombineField = 'ts',
payloadClass = 'com.example.MyPayload'
);
-- any payload class outside the built-in set lands the table on
-- hoodie.record.merge.mode=CUSTOM and the payload-based strategy id
insert into t values (1, parse_json('{"key":"value1"}'), 1000), (2, parse_json('{"key":"value2"}'), 1000);
update t set v = parse_json('{"key":"updated"}') where id = 1;
select id, cast(v as string) from t order by id;
```

**Expected behavior**

Variant reads on payload-based MOR tables return the same values as on the default mergers, whether or not the query lets Spark push the variant extraction into the scan.

**Possible direction**

Both sides of the merge and the reader output have to agree on one shape. For payload-based tables that probably means reading base and log rows as VariantVal, letting the avro round trip see the schema it expects, and applying the PushVariantIntoScan projection once on the reader output instead of inside the scan. Gating the base overlay on `shouldProjectVariants()` alone is not enough, because `appendPartitionAndProject` still expects the projected shape.

**Environment Description**

* Hudi version: master (1.3.0-SNAPSHOT)
* Spark version: 4.1 / 4.2 (PushVariantIntoScan)
* Storage: any
* Running on Docker: no

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with SparkFileFormatInternalRowReaderContext.getFileRecordIterator, HoodieAvroRecordMerger.merge, PayloadUpdateProcessor.handleNonDeletes, and HoodieFileGroupReaderBasedFileFormat.appendPartitionAndProject; compare the base and log row shapes when shouldProjectVariants() applies. Run the SQL reproduction under the spark4.1 or spark4.2 profile, then verify payload-based MOR reads return the expected variant values with and without pushed variant extraction.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.