apache / apache/hudi

Support async clustering w/ test suite job

Open
#14,746 0 comments 0 reactions 0 assignees View on GitHub
area:table-service area:tests component:clustering from-jira priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

As of now, we only have inline clustering support w/ hoodie test suite job. we need to add support for async clustering. 

This might be tricky since the regular writes should not overstep w/ clustering. if not the pipeline will fail. So, data generation has to go hand in hand w/ clustering configs. For eg, if clustering will get triggered every 4 commits, data generation should switch partitions for every 4 batches of input. That way there won't be any overstepping and pipeline can run for as many iterations as needed. 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1590
- Type: Test
- Epic: https://issues.apache.org/jira/browse/HUDI-3303
- Affects version(s):
- 0.9.0

---

## Comments

08/Mar/21 13:54;legendtkl;Hi, I would like to take this one. I have requested "assign to me", thanks.;;;

---

08/Mar/21 14:48;309637554;[~legendtkl] try it . :D;;;

---

27/Mar/22 14:41;xushiyan;[~legendtkl] did you get a chance to work on this?;;;

---

28/Mar/22 02:25;legendtkl;Hi, [~xushiyan] ,I'm very sorry about this. I am afraid that I don't have much time to work on this. Would you assign this to another guy who is interested about this?

 

Thanks. ;;;

---

28/Mar/22 03:39;xushiyan;[~legendtkl] no worries. ;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the hoodie test suite job and its existing inline clustering support, then trace how data generation and clustering configs are coordinated. Check the JIRA context for HUDI-1590 and HUDI-3303. Done means async clustering can run through repeated pipeline iterations without regular writes overstepping clustering.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.