[SUPPORT] RLI bootstrap unnecessarily persists+counts records to size file groups; use base file footer row counts instead
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
## Tips before filing an issue
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? Yes
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
## Describe the problem you faced
During Record Level Index (RLI) bootstrap, `HoodieBackedTableMetadataWriter#initializeRecordIndexPartition` reads record keys from the data table base files (and merged log files, for MOR) into an `HoodieData` and then calls `estimateFileGroupCount(records)`. The estimation supplier persists the RDD (`records.persist("MEMORY_AND_DISK_SER")`) and then iterates over the entire dataset (`records.count()`) just to obtain the total record count used to size the RLI file groups.
On large tables this is the primary latency and memory bottleneck of RLI bootstrap:
- The full materialized RDD of `(record key, location)` pairs is forced into the cluster's storage tier before any RLI commit work can start.
- Even though we already know the on-disk row counts (Parquet/ORC/HFile/Lance footers carry this), we pay an extra distributed scan over every row.
- When the user has pinned the RLI file group count (`min == max`), the count is not even needed — the estimation runs anyway.
## To Reproduce
Steps to reproduce the behavior:
1. Enable RLI on a sufficiently large existing table (e.g. via `hoodie.metadata.record.index.enable=true` or `hoodie.metadata.record.level.index.enable=true`).
2. Trigger metadata table initialization (any write to the data table will do so).
3. Observe in the driver logs that RLI initialization spends a significant fraction of time on a Spark stage that persists and counts the RLI records RDD before any file groups are written.
## Expected behavior
RLI bootstrap should estimate the total record count from already-available footer metadata of base files rather than materializing and counting an RDD of record entries. When `record.index.min.filegroup.count == record.index.max.filegroup.count` the estimation should be skipped entirely and the configured value used directly.
## Environment Description
- Hudi version : master (post 1.x)
- Spark version : any
- Hive version : N/A
- Hadoop version : N/A
- Storage (HDFS/S3/GCS..) : any
- Running on Docker? (yes/no) : no
## Additional context
This issue is filed to track the optimization of the RLI bootstrap path. A patch is forthcoming.
## Stacktrace
N/A (performance issue, not a crash).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at HoodieBackedTableMetadataWriter#initializeRecordIndexPartition and follow estimateFileGroupCount, focusing on where the RLI records RDD is persisted and counted. Check how base-file footer row counts are exposed for the supported formats. Done means bootstrap uses footer counts, skips estimation when the configured minimum and maximum file-group counts match, and no longer materializes and counts the full RDD for sizing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100