apache / apache/hudi

RLI bootstrap should compute RLI partitions using sampling

Open
#14,111 0 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**
Currently during bootstrap, number of RLI partitions are estimated. The issue aims to fix this by sampling the records instead of doing estimation.

**Why this task is needed:**
The issue will help in computing a better value for number of RLI partitions and thus more efficient partitioning of RLI records.

### Task Type

Code improvement/refactoring

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
https://github.com/apache/hudi/issues/14068
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the bootstrap path that currently estimates the number of RLI partitions, then review related issue 14068 for surrounding context. Done means bootstrap samples records instead of relying on estimation and produces a better RLI partition count for more efficient partitioning.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.