apache / apache/hudi

[SUPPORT] RLI bootstrap unnecessarily persists+counts records to size file groups; use base file footer row counts instead

Open
#18,825 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Tips before filing an issue

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? Yes

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

## Describe the problem you faced

During Record Level Index (RLI) bootstrap, `HoodieBackedTableMetadataWriter#initializeRecordIndexPartition` reads record keys from the data table base files (and merged log files, for MOR) into an `HoodieData` and then calls `estimateFileGroupCount(records)`. The estimation supplier persists the RDD (`records.persist("MEMORY_AND_DISK_SER")`) and then iterates over the entire dataset (`records.count()`) just to obtain the total record count used to size the RLI file groups.

On large tables this is the primary latency and memory bottleneck of RLI bootstrap:
- The full materialized RDD of `(record key, location)` pairs is forced into the cluster's storage tier before any RLI commit work can start.
- Even though we already know the on-disk row counts (Parquet/ORC/HFile/Lance footers carry this), we pay an extra distributed scan over every row.
- When the user has pinned the RLI file group count (`min == max`), the count is not even needed — the estimation runs anyway.

## To Reproduce

Steps to reproduce the behavior:

1. Enable RLI on a sufficiently large existing table (e.g. via `hoodie.metadata.record.index.enable=true` or `hoodie.metadata.record.level.index.enable=true`).
2. Trigger metadata table initialization (any write to the data table will do so).
3. Observe in the driver logs that RLI initialization spends a significant fraction of time on a Spark stage that persists and counts the RLI records RDD before any file groups are written.

## Expected behavior

RLI bootstrap should estimate the total record count from already-available footer metadata of base files rather than materializing and counting an RDD of record entries. When `record.index.min.filegroup.count == record.index.max.filegroup.count` the estimation should be skipped entirely and the configured value used directly.

## Environment Description

- Hudi version : master (post 1.x)
- Spark version : any
- Hive version : N/A
- Hadoop version : N/A
- Storage (HDFS/S3/GCS..) : any
- Running on Docker? (yes/no) : no

## Additional context

This issue is filed to track the optimization of the RLI bootstrap path. A patch is forthcoming.

## Stacktrace

N/A (performance issue, not a crash).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at HoodieBackedTableMetadataWriter#initializeRecordIndexPartition and follow estimateFileGroupCount, focusing on where the RLI records RDD is persisted and counted. Check how base-file footer row counts are exposed for the supported formats. Done means bootstrap uses footer counts, skips estimation when the configured minimum and maximum file-group counts match, and no longer materializes and counts the full RDD for sizing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.