apache / apache/hudi

Write tests for async clustering job to run concurrently w/ deltastreamer w/o overlap

Open
#15,123 0 comments 0 reactions 0 assignees View on GitHub
area:ingest area:table-service area:tests component:clustering from-jira priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

We want to test HoodieClusteringJob along w/ deltastreamer continuous job. But to test it reliably, we need to ensure regular writers don't overlap w/ clustering file groups. If not, one of them will fail. So, we need to come up w/ test data generation differently so that regular writers keep writing to new partition in every batch and so clustering can take care of older partitions and there won't be any over-stepping among these. 

 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3872
- Type: Test

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading HoodieClusteringJob and the Deltastreamer continuous job described in the issue, then inspect how their test data is currently generated. The work is complete when a reliable concurrent test writes to a new partition in each batch, lets clustering process older partitions, and demonstrates that regular writers and clustering do not overlap or fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.