apache / apache/hudi

Add tests to Multitable delta streamer for more sources and mix of diff sources

Open
#14,735 0 comments 0 reactions 0 assignees View on GitHub
area:ingest area:tests from-jira priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Add more tests to Multitable delta streamer for more sources and mix of diff sources.

Current tests cover only JsonKafkaSource and ParquetSource

We need more coverage
* AvroSoruce
* AvroKafkaSource
* CsvDFSSource
* HiveIncrPullSource
* HoodieIncrSource
* JsonDFSSource

I am not sure how much is testable. but worth covering as much as possible. 

Recently added tests for ParquetDFSSource: [https://github.com/apache/hudi/pull/2577.] Tried my best to templatize tests. Would be nice to re-use code as much as possible. 

Apart from adding tests for these sources in a homogenous manner, we also need tests in a heterogenous set up. Table1: source type1. Table2: source type2. 

For eg, ParquetDFSSource and CsvDFSSource. 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1619
- Type: Test
- Epic: https://issues.apache.org/jira/browse/HUDI-3303

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the existing Multitable delta streamer tests for JsonKafkaSource and ParquetSource, then review the ParquetDFSSource test approach in PR 2577. Extend the reusable tests for the listed sources and add heterogeneous combinations such as ParquetDFSSource with CsvDFSSource; done means the feasible sources and mixed-source cases have coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.