apache / apache/hudi

[Spark][LSM] Support clustering and bucket rescale

Open
#19,440 0 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**

Support safe built-in Spark clustering and bucket rescale operations for LSM-layout data tables.

**Why this task is needed:**

Clustering rewrites base files and may use custom-column, spatial-curve, binary-copy, or bucket-specific execution strategies. Any strategy that emits records in a non-record-key order breaks the LSM sorted-run invariant.

### Implementation Scope

- Support the built-in Spark sort-and-size and single-file clustering strategies for COW and MOR LSM tables.
- Reuse table-aware LSM RDD/Row partitioners when clustering has no conflicting user sort.
- Support consistent-bucket clustering and consistent-hashing split/merge (`BUCKET_RESCALE`) while preserving bucket routing and file-id semantics.
- For consistent-hashing operations:
- Split an ordered input and retain ordered subsequences per output bucket.
- K-way merge multiple sorted input file groups before routing a bucket merge; do not concatenate overlapping sorted runs.
- Disable binary/stream-copy optimization for LSM tables and fall back to an ordered rewrite.
- Fail before clustering becomes inflight for custom sort columns, Z-order, Hilbert, or unknown custom execution strategies whose ordering cannot be verified.
- Keep clustering replace-commit and input file-group replacement semantics unchanged.

### Acceptance Criteria

- [ ] Built-in RDD and Row clustering produces format-aware sorted output files.
- [ ] COW and MOR snapshots remain correct after clustering and after a subsequent upsert.
- [ ] Binary/stream-copy strategies fall back to a sorted rewrite for LSM tables.
- [ ] Consistent-hashing split and merge preserve routing, file IDs, and sorted output.
- [ ] Conflicting or unverifiable strategies fail before inflight.
- [ ] Replace-commit metadata contains the expected replaced file groups.
- [ ] Targeted clustering tests, Spark DataSource E2E, compile, checkstyle/scalastyle, and `git diff --check` pass.

### Related Issues

**Parent feature issue:** #19065

**Depends on:** #19436, #19438, #19439

**Out of scope:** bootstrap and metadata-table index operations.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the existing Spark RDD and Row clustering paths, LSM table partitioners, and consistent-bucket or consistent-hashing rescale handling described in the task. Run the targeted clustering tests and Spark DataSource E2E checks first. Done means sorted LSM output, correct COW and MOR snapshots after upsert, preserved routing and file IDs, and early failure for unverifiable strategies.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.