apache / apache/hudi

[to be discussed] Add support for clustering earlier instant times first

Open
#17,956 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**
Add a new clustering config where, if enabled, `org.apache.hudi.table.action.cluster.strategy.PartitionAwareClusteringPlanStrategy#buildClusteringGroupsForPartition` will target earlier instant times in a partition first (if not all files in a partition can be clustered). For example, we can change from sorting input data files by `(-1 * file size)` to instead `(instant time, -1 * file size)` when building clustering groups.

**Why this feature is needed:**
We have use cases for bulk-insert datasets we run clustering on the latest few partitions to "stitch" together small files. But our jobs do not have sufficient resources to target all files in the partition in a short enough time. In this scenario, we want to prioritize earlier files in a partition to be stitched together first, since we expect queries to target data in those files first. We have implemented the suggested implementaiton internally the above to achieve this guarantee. Although this is less optimal in terms of "packing files", it is fine for our use case as we typically have hundreds or thousands of files for each instant time.

### User Experience

**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples

### Hudi RFC Requirements

**RFC PR link:** (if applicable)

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with org.apache.hudi.table.action.cluster.strategy.PartitionAwareClusteringPlanStrategy#buildClusteringGroupsForPartition and inspect how input files are currently sorted. Trace the clustering configuration and related tests, then clarify the required configuration and API impact. Done means the enabled behavior prioritizes earlier instant times within a partition while preserving the stated file-size ordering, with coverage for partial partition clustering.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.