Write tests for async clustering job to run concurrently w/ deltastreamer w/o overlap
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
We want to test HoodieClusteringJob along w/ deltastreamer continuous job. But to test it reliably, we need to ensure regular writers don't overlap w/ clustering file groups. If not, one of them will fail. So, we need to come up w/ test data generation differently so that regular writers keep writing to new partition in every batch and so clustering can take care of older partitions and there won't be any over-stepping among these.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3872
- Type: Test
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading HoodieClusteringJob and the Deltastreamer continuous job described in the issue, then inspect how their test data is currently generated. The work is complete when a reliable concurrent test writes to a new partition in each batch, lets clustering process older partitions, and demonstrates that regular writers and clustering do not overlap or fail.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100