apache / apache/hudi

[to be discussed] Support for rollbackFailedWrites to delete inactive clustering plans

Open
#17,879 3 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**
A new `HoodieWriteConfig` key should be added to indicate that any incomplete clustering plans with an expired heartbeat should be completely rolled back by `rollbackFailedWrites`, with the plan deleted. HUDI will assume that clustering plans are only be attempted once (by the same writer client that scheduled the plan).

Specifically, HUDI should do the following if and only if this config key is enabled
- `scheduleClustering` should start a heartbeat for determined instant time before publishing the plan to timeline. It should not start the heartbeat when executing the clustering plan in `cluster`
- `rollbackFailedWrites` should schedule and execute rollback plan of any incomplete clustering instants with expired heartbeat.
- provide a new utility API `rollbackClusteringWithMatchingPartitions` that takes a list of partitions and attempts a rollback of any incomplete clustering plans targeting the same partition (provided they have expired heartbeat)

Note that we are making the following assumptions
- The value of this config key is the same across all writers to a dataset (given a multi-writer setup)
- The writer client that schedules a clustering plan will then immediately execute the clustering plan

**Why this task is needed**
For our datasets, we do not perform clustering in the same writer job as ingestion writes. Rather, we orchestrate separate service that attempt clustering. In addition, we can have concurrent jobs attempt to cluster the same datasets with different types of clustering plans (for example, we may stitch files in newer and older partitions with different strategies). Given this, we had to implement the above requirements in our internal HUDI 0.x build in order to ensure:
- if we cannot re-execute an existing clustering plan (due more spark resources that are currently available), then we can ensure these inflight plans are rolled back without blocking archival/clean/metadata table services. Since for us clustering typically has a lower priority than other operations.
- our clustering service job must be able to rollback any existing "leftover" clustering plans targeting the same partition before proceeding to create a new clustering plan. This is since when HUDI schedules a clustering plan, it will ignore files already targeted by other incomplete clustering plans. Since the current job may not have sufficient spark resourcing to re-attempt the plan, and the execution of this "leftover" plan may not anyway be of high priority.

Once we achieve consensus on these requirements, we can start upstreaming our implementations

### Task Type

Code improvement/refactoring

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing scheduleClustering, cluster, and rollbackFailedWrites, then review HoodieWriteConfig and the clustering rollback flow. The requirements still need consensus before implementation. Done means the new configuration controls heartbeat creation, expired incomplete clustering plans are rolled back and deleted, and rollbackClusteringWithMatchingPartitions handles matching partitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.