apache / apache/seatunnel

[Umbrella][Feature][Connector] Need help for supporting multi-table source feature

Open
#10,425 27 comments 0 reactions 0 assignees View on GitHub
feature good first issue help wanted
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

Based on the [tracking issue #5677](https://github.com/apache/seatunnel/issues/5677) and source code analysis, the following connectors have not yet implemented true multi-table support (i.e., missing `tables_configs` (or deprecated `table_list`) configuration or the ability to handle multiple distinct tables/schemas in a single source instance).

We need community help to implement these!

### Connectors to be implemented:

| Type | Connector | Contributer | Status | PR |
| :--- | :--- | :--- | :--- | :--- |
| Source | AmazonDynamoDB | @vbhavh | Doing | |
| Source | Cassandra | @NixonWahome | Done | #10896 |
| Source | Elasticsearch | @FuYouJ | Doing | |
| Source | Hudi | @LiJie20190102 | Doing | |
| Source | InfluxDB | | Todo | |
| Source | IoTDB | | Todo | |
| Source | MongoDB (Batch) | @chl-wxp | Doing | |
| Source | Neo4j | | Todo | |
| Source | OpenMldb | | Todo | |
| Source | Pulsar | @onceMisery | Done | [#10825](https://github.com/apache/seatunnel/pull/10825) |
| Source | RabbitMQ | @Anthippi | Doing | [#10557](https://github.com/apache/seatunnel/pull/10557) |
| Source | Redis | @JeremyXin | Done | [#10601](https://github.com/apache/seatunnel/pull/10601) |
| Source | RocketMQ | @kuleat | Done | [#10619](https://github.com/apache/seatunnel/pull/10619) |

### Reference

- **Feature Issue**: #5677
- **Implementation References**:
- `JdbcSource` and `JdbcSourceFactory` (Supports `TABLE_LIST` - Note: newer implementations should refer to `TABLE_CONFIGS`)
- `KafkaSource` (Supports `TABLE_LIST`)
- `PaimonSource` (Supports `CatalogOptions.TABLE_LIST`)
- **Standard**: Please use `ConnectorCommonOptions.TABLE_CONFIGS` (`tables_configs`) as the configuration key, as `CatalogOptions.TABLE_LIST` (`table_list`) is deprecated.

### How to contribute

1. Pick a connector from the list.
2. Search [Pull Requests](https://github.com/apache/seatunnel/pulls) to ensure no one is already working on it (keywords: "multi table", "ConnectorName").
3. Comment on this issue to state which connector you'd like to help implement.
4. Implement `SupportMultipleTable` logic (usually involves adding `OptionRule` for `TABLE_CONFIGS` and updating the Source to handle multiple `CatalogTable`s).
5. Submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Choose an unclaimed connector marked Todo, first checking related pull requests and commenting on this issue. Read its source and factory alongside JdbcSource, JdbcSourceFactory, KafkaSource, or PaimonSource, and review tracking issue #5677. Done means the connector supports multiple distinct tables or schemas using ConnectorCommonOptions.TABLE_CONFIGS and has a submitted PR.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.