apache / apache/hudi

Deferred RLI init for fresh tables fails to bootstrap on 2nd commit when write schema is unset

Open
#18,866 0 comments 0 reactions 0 assignees View on GitHub
area:index area:metadata-table type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Tips before filing an issue

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

### Describe the problem you faced

Follow-up to #18353 which introduced `hoodie.metadata.record.level.index.defer.init` to defer Record Level Index (RLI) bootstrap to the 2nd commit on a fresh table. The current implementation resolves the data schema used to read base/log files during RLI init via `HoodieSchema.parse(dataWriteConfig.getWriteSchema())`. The write config schema is not always populated when the metadata writer is constructed in the deferred path:

- On the 2nd commit (after a deferred first commit), the write config used to build the metadata writer may not carry the avro schema string.
- When the metadata writer is constructed outside an active write (e.g. read-side helpers building a `metadataWriter(writeConfig)`).

In those cases the inline parse fails (NPE / empty schema parse) and the deferred RLI bootstrap on commit #2 breaks. The bulk_insert path on a fresh table with defer enabled hits this consistently.

### To Reproduce

Steps to reproduce the behavior:

1. Create a fresh Hudi MoR table with `hoodie.metadata.record.level.index.defer.init=true` and RLI enabled (partitioned).
2. Issue commit #1 as `bulk_insert` (Overwrite). RLI is correctly deferred — partition is not initialized.
3. Issue commit #2 as `bulk_insert` (Append). On the 2nd metadata-writer entry, deferred RLI init runs, but the schema resolution via `dataWriteConfig.getWriteSchema()` is empty in this path, and RLI bootstrap fails.

### Expected behavior

The deferred RLI partition should bootstrap successfully on the 2nd commit (or on any subsequent metadata writer construction once there is at least one completed instant on the data table), regardless of whether the write config carries an explicit avro schema string. The latest committed table schema is a safe fallback.

### Environment Description

- Hudi version: master (1.2.0-rc2 line)
- Spark version: 3.x
- Storage (HDFS/S3/GCS..): n/a
- Running on Docker? (yes/no): no

### Additional context

- Original feature PR: #18353
- Affected code path: `HoodieBackedTableMetadataWriter#initializeFromFilesystem` → `initializeFilegroupsAndCommitToRecordIndexPartition` → `readRecordKeysFromFileSliceSnapshot`, where the data schema is parsed from `dataWriteConfig.getWriteSchema()`.

### Stacktrace

`Cannot initialize record level index` / NPE from `HoodieSchema.parse(null)` inside the executor closure of `readRecordKeysFromFileSliceSnapshot` when the deferred RLI bootstrap is triggered with an unset write schema.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at HoodieBackedTableMetadataWriter#initializeFromFilesystem and follow initializeFilegroupsAndCommitToRecordIndexPartition into readRecordKeysFromFileSliceSnapshot. Reproduce with a fresh partitioned Hudi MoR table using deferred RLI, then run bulk_insert commits #1 and #2 with the write schema unset. Done means deferred RLI bootstrap succeeds on the second or later metadata-writer construction using the latest committed table schema as fallback.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.