[BUG] Native delete log schema rejects nested ordering fields on table version 10
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
`HoodieSchemaUtils.createDeleteLogSchema` (`HoodieSchemas.createDeleteLogSchema` before #19809) resolves each ordering field with `tableSchema.getField(name)`, an exact top-level lookup (`HoodieSchema.java:1215`). A nested ordering field such as `nested_record.level` is never a top-level name, so the call throws `IllegalArgumentException: Ordering field nested_record.level not found in table schema`.
Nested precombine/ordering fields are supported (HUDI-4051, #5517) and `HoodieRecordUtils.getOrderingFieldNames` passes `hoodie.table.ordering.fields` through verbatim, so on a table version 10 MOR table with `EVENT_TIME_ORDERING` and a nested ordering field:
- write: `HoodieNativeLogAppendHandle.createLogWriterForAppend` -> `HoodieNativeLogFormatWriter.appendDeleteRecord` -> `ensureDeleteFileWriter` (`HoodieNativeLogFormatWriter.java:261`) throws on the first delete.
- read: `HoodieNativeLogDeleteBlock` (`:72`) and `LsmFileIterators.createNativeDeleteLogIterator` (`:176`) throw for any native delete log written under such a table config.
Present since #18987 / #19072; reachable by default since #19118 made native logs the default for table version >= 10. Not a regression from #19809, which moves the method unchanged. No in-repo test covers a nested ordering field on a MOR table (`TestFileGroupReaderPartitionColumn.scala:192` uses `nested_record.level` on COW only).
Minimal reproduction:
```java
HoodieSchema tableSchema = ...; // record with field nested_record: record { level: long }
HoodieSchemaUtils.createDeleteLogSchema(tableSchema, Collections.singletonList("nested_record.level"));
// IllegalArgumentException: Ordering field nested_record.level not found in table schema
```
Fix direction: resolve the field via `HoodieSchema#getNestedField` and give the delete-log column a flat name (Avro field names cannot contain `.`), then align the lookups that use the dotted name against the flat delete record: `HoodieNativeLogDeleteBlock.java:113,129` and `LsmFileIterators.createNativeDeleteRecord` call `getOrderingValue(record, deleteLogSchema, orderingFieldNames)`. `HoodieNativeLogFormatWriter.createDeleteLogFieldValues` is positional and needs no change.
### Environment
- Hudi master (`c59987a024cd`), table version 10, MOR, native log format (default)
- `hoodie.record.merge.mode=EVENT_TIME_ORDERING` with a nested ordering field, any engine
### Logs and Stack Trace
```
java.lang.IllegalArgumentException: Ordering field nested_record.level not found in table schema
at org.apache.hudi.common.schema.HoodieSchemaUtils.lambda$createDeleteLogSchema$...
at org.apache.hudi.io.cdc.HoodieNativeLogFormatWriter.ensureDeleteFileWriter(HoodieNativeLogFormatWriter.java:261)
at org.apache.hudi.io.cdc.HoodieNativeLogFormatWriter.appendDeleteRecord(HoodieNativeLogFormatWriter.java:169)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with HoodieSchemaUtils.createDeleteLogSchema and trace the native delete-log write path through HoodieNativeLogFormatWriter, then inspect HoodieNativeLogDeleteBlock.java:113,129 and LsmFileIterators.createNativeDeleteRecord. Compare the nested-field handling with TestFileGroupReaderPartitionColumn.scala:192 and reproduce the table-version-10 MOR case. Done means nested ordering fields work for native delete-log writes and reads, with regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100