Add tests to Multitable delta streamer for more sources and mix of diff sources
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Add more tests to Multitable delta streamer for more sources and mix of diff sources.
Current tests cover only JsonKafkaSource and ParquetSource
We need more coverage
* AvroSoruce
* AvroKafkaSource
* CsvDFSSource
* HiveIncrPullSource
* HoodieIncrSource
* JsonDFSSource
I am not sure how much is testable. but worth covering as much as possible.
Recently added tests for ParquetDFSSource: [https://github.com/apache/hudi/pull/2577.] Tried my best to templatize tests. Would be nice to re-use code as much as possible.
Apart from adding tests for these sources in a homogenous manner, we also need tests in a heterogenous set up. Table1: source type1. Table2: source type2.
For eg, ParquetDFSSource and CsvDFSSource.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1619
- Type: Test
- Epic: https://issues.apache.org/jira/browse/HUDI-3303
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the existing Multitable delta streamer tests for JsonKafkaSource and ParquetSource, then review the ParquetDFSSource test approach in PR 2577. Extend the reusable tests for the listed sources and add heterogeneous combinations such as ParquetDFSSource with CsvDFSSource; done means the feasible sources and mixed-source cases have coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100