apache / apache/hudi

Parallelize deduplication in CLI tool

Open
#15,484 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:medium type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The CLI tool command `repair deduplicate` repair one partition at a time.  To repair hundreds of partitions, this takes time.  We should add a mode to take multiple partition paths for the CLI and run the dedup job for multiple partitions at the same time.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4990
- Type: Improvement

---

## Comments

09/Nov/22 18:35;jonvex;We are stuck on this for now because we can't run the integration tests locally;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the CLI implementation of the `repair deduplicate` command and trace how it accepts a partition path and runs the deduplication job. The change is complete when the command accepts multiple partition paths and runs deduplication for them concurrently; the issue notes that integration tests cannot currently be run locally.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.