Support async clustering w/ test suite job
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
As of now, we only have inline clustering support w/ hoodie test suite job. we need to add support for async clustering.
This might be tricky since the regular writes should not overstep w/ clustering. if not the pipeline will fail. So, data generation has to go hand in hand w/ clustering configs. For eg, if clustering will get triggered every 4 commits, data generation should switch partitions for every 4 batches of input. That way there won't be any overstepping and pipeline can run for as many iterations as needed.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1590
- Type: Test
- Epic: https://issues.apache.org/jira/browse/HUDI-3303
- Affects version(s):
- 0.9.0
---
## Comments
08/Mar/21 13:54;legendtkl;Hi, I would like to take this one. I have requested "assign to me", thanks.;;;
---
08/Mar/21 14:48;309637554;[~legendtkl] try it . :D;;;
---
27/Mar/22 14:41;xushiyan;[~legendtkl] did you get a chance to work on this?;;;
---
28/Mar/22 02:25;legendtkl;Hi, [~xushiyan] ,I'm very sorry about this. I am afraid that I don't have much time to work on this. Would you assign this to another guy who is interested about this?
Thanks. ;;;
---
28/Mar/22 03:39;xushiyan;[~legendtkl] no worries. ;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the hoodie test suite job and its existing inline clustering support, then trace how data generation and clustering configs are coordinated. Check the JIRA context for HUDI-1590 and HUDI-3303. Done means async clustering can run through repeated pipeline iterations without regular writes overstepping clustering.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100