apache / apache/seatunnel

[Question] How to write a configuration file for merging feature levels of different data sources and tables?

Open
#10,147 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

How to write a configuration file for merging feature levels of different data sources and tables? Can it be achieved at present? If it's not possible, can you help me explain the reason? If possible, please help me write a related configuration file. For example, the tables in the two data sources have the same field ID number, but they have different attribute fields. So I want to combine the different attribute fields of the data with the same ID number into one, and then write them into the table of the target data source.

### SeaTunnel Version

2.312

### SeaTunnel Config

```conf
This is the configuration of different tables from the same source:
"{\"source\":[{\"plugin_name\":\"JDBC\",\"plugin_output\":\"fake\",\"url\":\"jdbc:mysql://192.168.8.25:3306/test\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"root\",\"password\":\"123456\",\"query\":\"select UUID() as id,table1.name,table1.age,table1.address,table1.id_card_number,table2.phone from table1 LEFT JOIN table2 on table1.id_card_number = table2.id_card_number\"}],\"sink\":[{\"plugin_name\":\"JDBC\",\"plugin_input\":\"fake\",\"url\":\"jdbc:postgresql://192.168.8.127:54321/test\",\"driver\":\"org.postgresql.Driver\",\"user\":\"yid_blc_mds\",\"password\":\"Yuyao123456\",\"query\":\"INSERT INTO table_result (id, name, age,address, id_card_number, phone) VALUES (?, ?, ?, ?, ?, ?)\"}]}","env": "{\"parallelism\":2,\"job.mode\":\"BATCH\",\"checkpoint.interval\":10000}"
```

### Running Command

```shell
This is the configuration of different tables from the same source:
"{\"source\":[{\"plugin_name\":\"JDBC\",\"plugin_output\":\"fake\",\"url\":\"jdbc:mysql://192.168.8.25:3306/test\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"root\",\"password\":\"123456\",\"query\":\"select UUID() as id,table1.name,table1.age,table1.address,table1.id_card_number,table2.phone from table1 LEFT JOIN table2 on table1.id_card_number = table2.id_card_number\"}],\"sink\":[{\"plugin_name\":\"JDBC\",\"plugin_input\":\"fake\",\"url\":\"jdbc:postgresql://192.168.8.127:54321/test\",\"driver\":\"org.postgresql.Driver\",\"user\":\"yid_blc_mds\",\"password\":\"Yuyao123456\",\"query\":\"INSERT INTO table_result (id, name, age,address, id_card_number, phone) VALUES (?, ?, ?, ?, ?, ?)\"}]}","env": "{\"parallelism\":2,\"job.mode\":\"BATCH\",\"checkpoint.interval\":10000}"
```

### Error Exception

```log
no
```

### Zeta or Flink or Spark Version

_No response_

### Java or Scala Version

_No response_

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the JDBC source query and PostgreSQL sink INSERT configuration shown in the issue, then determine whether SeaTunnel supports merging rows from different tables or data sources by a shared ID. Done would require a confirmed supported configuration or a documented explanation that this workflow is not currently possible.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, postgresql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.