apache / apache/hudi

[BUG] Native delete log schema rejects nested ordering fields on table version 10

Open
#19,823 5 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

`HoodieSchemaUtils.createDeleteLogSchema` (`HoodieSchemas.createDeleteLogSchema` before #19809) resolves each ordering field with `tableSchema.getField(name)`, an exact top-level lookup (`HoodieSchema.java:1215`). A nested ordering field such as `nested_record.level` is never a top-level name, so the call throws `IllegalArgumentException: Ordering field nested_record.level not found in table schema`.

Nested precombine/ordering fields are supported (HUDI-4051, #5517) and `HoodieRecordUtils.getOrderingFieldNames` passes `hoodie.table.ordering.fields` through verbatim, so on a table version 10 MOR table with `EVENT_TIME_ORDERING` and a nested ordering field:

- write: `HoodieNativeLogAppendHandle.createLogWriterForAppend` -> `HoodieNativeLogFormatWriter.appendDeleteRecord` -> `ensureDeleteFileWriter` (`HoodieNativeLogFormatWriter.java:261`) throws on the first delete.
- read: `HoodieNativeLogDeleteBlock` (`:72`) and `LsmFileIterators.createNativeDeleteLogIterator` (`:176`) throw for any native delete log written under such a table config.

Present since #18987 / #19072; reachable by default since #19118 made native logs the default for table version >= 10. Not a regression from #19809, which moves the method unchanged. No in-repo test covers a nested ordering field on a MOR table (`TestFileGroupReaderPartitionColumn.scala:192` uses `nested_record.level` on COW only).

Minimal reproduction:

```java
HoodieSchema tableSchema = ...; // record with field nested_record: record { level: long }
HoodieSchemaUtils.createDeleteLogSchema(tableSchema, Collections.singletonList("nested_record.level"));
// IllegalArgumentException: Ordering field nested_record.level not found in table schema
```

Fix direction: resolve the field via `HoodieSchema#getNestedField` and give the delete-log column a flat name (Avro field names cannot contain `.`), then align the lookups that use the dotted name against the flat delete record: `HoodieNativeLogDeleteBlock.java:113,129` and `LsmFileIterators.createNativeDeleteRecord` call `getOrderingValue(record, deleteLogSchema, orderingFieldNames)`. `HoodieNativeLogFormatWriter.createDeleteLogFieldValues` is positional and needs no change.

### Environment

- Hudi master (`c59987a024cd`), table version 10, MOR, native log format (default)
- `hoodie.record.merge.mode=EVENT_TIME_ORDERING` with a nested ordering field, any engine

### Logs and Stack Trace

```
java.lang.IllegalArgumentException: Ordering field nested_record.level not found in table schema
at org.apache.hudi.common.schema.HoodieSchemaUtils.lambda$createDeleteLogSchema$...
at org.apache.hudi.io.cdc.HoodieNativeLogFormatWriter.ensureDeleteFileWriter(HoodieNativeLogFormatWriter.java:261)
at org.apache.hudi.io.cdc.HoodieNativeLogFormatWriter.appendDeleteRecord(HoodieNativeLogFormatWriter.java:169)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HoodieSchemaUtils.createDeleteLogSchema and trace the native delete-log write path through HoodieNativeLogFormatWriter, then inspect HoodieNativeLogDeleteBlock.java:113,129 and LsmFileIterators.createNativeDeleteRecord. Compare the nested-field handling with TestFileGroupReaderPartitionColumn.scala:192 and reproduce the table-version-10 MOR case. Done means nested ordering fields work for native delete-log writes and reads, with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.