[Umbrella][Feature][Connector] Need help for supporting multi-table source feature
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 204
Description
Based on the [tracking issue #5677](https://github.com/apache/seatunnel/issues/5677) and source code analysis, the following connectors have not yet implemented true multi-table support (i.e., missing `tables_configs` (or deprecated `table_list`) configuration or the ability to handle multiple distinct tables/schemas in a single source instance).
We need community help to implement these!
### Connectors to be implemented:
| Type | Connector | Contributer | Status | PR |
| :--- | :--- | :--- | :--- | :--- |
| Source | AmazonDynamoDB | @vbhavh | Doing | |
| Source | Cassandra | @NixonWahome | Done | #10896 |
| Source | Elasticsearch | @FuYouJ | Doing | |
| Source | Hudi | @LiJie20190102 | Doing | |
| Source | InfluxDB | | Todo | |
| Source | IoTDB | | Todo | |
| Source | MongoDB (Batch) | @chl-wxp | Doing | |
| Source | Neo4j | | Todo | |
| Source | OpenMldb | | Todo | |
| Source | Pulsar | @onceMisery | Done | [#10825](https://github.com/apache/seatunnel/pull/10825) |
| Source | RabbitMQ | @Anthippi | Doing | [#10557](https://github.com/apache/seatunnel/pull/10557) |
| Source | Redis | @JeremyXin | Done | [#10601](https://github.com/apache/seatunnel/pull/10601) |
| Source | RocketMQ | @kuleat | Done | [#10619](https://github.com/apache/seatunnel/pull/10619) |
### Reference
- **Feature Issue**: #5677
- **Implementation References**:
- `JdbcSource` and `JdbcSourceFactory` (Supports `TABLE_LIST` - Note: newer implementations should refer to `TABLE_CONFIGS`)
- `KafkaSource` (Supports `TABLE_LIST`)
- `PaimonSource` (Supports `CatalogOptions.TABLE_LIST`)
- **Standard**: Please use `ConnectorCommonOptions.TABLE_CONFIGS` (`tables_configs`) as the configuration key, as `CatalogOptions.TABLE_LIST` (`table_list`) is deprecated.
### How to contribute
1. Pick a connector from the list.
2. Search [Pull Requests](https://github.com/apache/seatunnel/pulls) to ensure no one is already working on it (keywords: "multi table", "ConnectorName").
3. Comment on this issue to state which connector you'd like to help implement.
4. Implement `SupportMultipleTable` logic (usually involves adding `OptionRule` for `TABLE_CONFIGS` and updating the Source to handle multiple `CatalogTable`s).
5. Submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Choose an unclaimed connector marked Todo, first checking related pull requests and commenting on this issue. Read its source and factory alongside JdbcSource, JdbcSourceFactory, KafkaSource, or PaimonSource, and review tracking issue #5677. Done means the connector supports multiple distinct tables or schemas using ConnectorCommonOptions.TABLE_CONFIGS and has a submitted PR.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100