apache / apache/hudi

SQL UPDATE on MOR table with hoodie.populate.meta.fields=false writes a log file with an empty fileId

Open
#19,448 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Problem

A SQL `UPDATE` on a MOR table created with `hoodie.populate.meta.fields = 'false'` fails: the append handle writes a log file whose name has an empty fileId segment, which `HoodieLogFile.parseFieldsFromPath` then rejects.

### Repro (master, spark3.5 profile; taken verbatim from the failing test, not minimized)

```sql
CREATE TABLE t USING HUDI
PARTITIONED BY (part)
TBLPROPERTIES (
type = 'mor',
primaryKey = 'id',
orderingFields = 'ts',
hoodie.populate.meta.fields = 'false'
)
LOCATION '/tmp/t'
AS SELECT
1 AS id,
named_struct('name', 'a1', 'price', 10) AS item,
array(named_struct('k', 'k0', 'v', 'v0')) AS tags,
map('m0', named_struct('a', 1, 'b', 2)) AS props,
123456 AS ts,
'p1' AS part;

UPDATE t SET ts = 123457 WHERE id = 1;
```

fails with

```
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0
Caused by: org.apache.hudi.exception.InvalidHoodiePathException: Invalid path
file:/.../_0-106-150_20260731123923502_1.log.parquet of type LogFile
at org.apache.hudi.common.model.HoodieLogFile.parseFieldsFromPath(HoodieLogFile.java:129)
```

Note the log file name starts with `_` -- the fileId segment is empty. The same statements succeed with meta fields enabled.

### Expected

Either the update works with virtual keys, or the write fails fast with a clear unsupported-operation error instead of writing an unparseable log file path.

Found while extending `TestNestedSchemaPruningOptimization` in #19406.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the failing test in TestNestedSchemaPruningOptimization and reproduce the SQL UPDATE using the spark3.5 profile. Trace the append handle output through HoodieLogFile.parseFieldsFromPath, focusing on how the fileId is formed when hoodie.populate.meta.fields is false. Done means the update succeeds with virtual keys or fails fast with a clear unsupported-operation error, without writing an unparseable log path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.