SQL UPDATE on MOR table with hoodie.populate.meta.fields=false writes a log file with an empty fileId
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Problem
A SQL `UPDATE` on a MOR table created with `hoodie.populate.meta.fields = 'false'` fails: the append handle writes a log file whose name has an empty fileId segment, which `HoodieLogFile.parseFieldsFromPath` then rejects.
### Repro (master, spark3.5 profile; taken verbatim from the failing test, not minimized)
```sql
CREATE TABLE t USING HUDI
PARTITIONED BY (part)
TBLPROPERTIES (
type = 'mor',
primaryKey = 'id',
orderingFields = 'ts',
hoodie.populate.meta.fields = 'false'
)
LOCATION '/tmp/t'
AS SELECT
1 AS id,
named_struct('name', 'a1', 'price', 10) AS item,
array(named_struct('k', 'k0', 'v', 'v0')) AS tags,
map('m0', named_struct('a', 1, 'b', 2)) AS props,
123456 AS ts,
'p1' AS part;
UPDATE t SET ts = 123457 WHERE id = 1;
```
fails with
```
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0
Caused by: org.apache.hudi.exception.InvalidHoodiePathException: Invalid path
file:/.../_0-106-150_20260731123923502_1.log.parquet of type LogFile
at org.apache.hudi.common.model.HoodieLogFile.parseFieldsFromPath(HoodieLogFile.java:129)
```
Note the log file name starts with `_` -- the fileId segment is empty. The same statements succeed with meta fields enabled.
### Expected
Either the update works with virtual keys, or the write fails fast with a clear unsupported-operation error instead of writing an unparseable log file path.
Found while extending `TestNestedSchemaPruningOptimization` in #19406.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the failing test in TestNestedSchemaPruningOptimization and reproduce the SQL UPDATE using the spark3.5 profile. Trace the append handle output through HoodieLogFile.parseFieldsFromPath, focusing on how the fileId is formed when hoodie.populate.meta.fields is false. Done means the update succeeds with virtual keys or fails fast with a clear unsupported-operation error, without writing an unparseable log path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100