Improve hudi-utilities tests with KafkaTestUtils to allow more partitions
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
original discussion https://github.com/apache/hudi/pull/8445#discussion_r1203794976
We may want to figure out how to make the KafkaTestUtils work with more than 2 kafka partitions. it's limited to 2 somehow, prob. due to some internal spark partition limit.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6266
- Type: Test
- Epic: https://issues.apache.org/jira/browse/HUDI-3303
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the original discussion in PR 8445 and locate KafkaTestUtils in the hudi-utilities tests. Reproduce the limitation with more than two Kafka partitions, then verify that the relevant tests support additional partitions without regressing existing coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, stream-processing, testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100