[to be discussed] Spark clustering planner should support skipping clustering groups where number of input and output file slices are same
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
**What the feature achieves:**
Add a new clustering config, where if enabled, `org.apache.hudi.table.action.cluster.strategy.PartitionAwareClusteringPlanStrategy#buildClusteringGroupsForPartition` will not emit a clustering group if the number of input and output file slices are the same.
**Why this feature is needed:**
For our use case we use an execution class similar to `org.apache.hudi.client.clustering.run.strategy.SparkBinaryCopyClusteringExecutionStrategy` as a clustering execution class to merge smaller files together into one large file. We set `hoodie.clustering.plan.strategy.target.file.max.bytes` to the target output file size value. But there are scenarios where clustering groups with just one input & output file slice are created. For example, if a partition has these files
```
file1: 800 MB
file2: 800 MB
file3: 200 MB
```
and we cluster with target file size/clustering group size of 1 GB, then we will emit clustering groups
`([file1], 1), ([file2, file3], 2)` and we want to not execute the first group.
Adding the above config would achieve this.
We can upstream our internal implementation once we reach consensus
### User Experience
**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with org.apache.hudi.table.action.cluster.strategy.PartitionAwareClusteringPlanStrategy#buildClusteringGroupsForPartition and trace how clustering configuration reaches group creation. Review SparkBinaryCopyClusteringExecutionStrategy for the intended execution context. Done means consensus on the new configuration and API/RFC requirements, with groups skipped when input and output file-slice counts match.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100