[Bug] [Postgres-CDC] Failed to synchronize partitioned tables to MySQL using Postgres-CDC
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 203
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
我通过/hazelcast/rest/maps/submit-job进行提交任务,现在测试阶段打算把一个Postgres的分区表(pg_partman自动分区)的数据以流处理的方式同步到MySQL中,hocon配置如下
```hocon
env {
job.mode = "STREAMING"
execution.parallelism = 1
checkpoint.interval = 10000
}
source {
Postgres-CDC {
url = "jdbc:postgresql://172.31.204.230:5432/datahub"
username = "datahub"
password = "1111"
database-names = ["datahub"]
schema-names = ["original"]
table-names = ["datahub.ods.lazada_order_details"]
startup.mode = "latest"
slot.name = "seatunnel_slot"
decoding.plugin.name = "pgoutput"
}
}
transform {
}
sink {
Jdbc {
url = "jdbc:mysql://172.31.204.227:3306/shop_fox_prod?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true&allowPublicKeyRetrieval=true&nullCatalogMeansCurrent=true"
driver = "com.mysql.cj.jdbc.Driver"
username = "root"
password = "111"
generate_sink_sql = true
database = "shop_fox_prod"
table = "lazada_order_details"
primary_keys = ["id", "created_at"]
}
}
```
在web界面能看到任务长期处于RUNNING状态,但是source的TableSourceReceivedCount字段始终为0,sink的TableSinkWriteCount字段也为0,目前使用的是apache/seatunnel:2.3.13的docker镜像,默认使用Zeta引擎,
然后观察日志发现有注意到:SELECT reltuples FROM pg_class r WHERE relkind = 'r' AND relname = 'lazada_order_details'; 的SQL,是因为现在的Postgres-CDC只能同步普通表吗?
### SeaTunnel Version
2.3.13
### SeaTunnel Config
```conf
env {
job.mode = "STREAMING"
execution.parallelism = 1
checkpoint.interval = 10000
}
source {
Postgres-CDC {
url = "jdbc:postgresql://172.31.204.230:5432/datahub"
username = "datahub"
password = "TeKvkjtFtBceyCF6"
database-names = ["datahub"]
schema-names = ["original"]
table-names = ["datahub.original.lazada_order_details"]
startup.mode = "latest"
slot.name = "seatunnel_slot"
decoding.plugin.name = "pgoutput"
}
}
transform {
}
sink {
Jdbc {
url = "jdbc:mysql://172.31.204.227:3306/shop_fox_prod?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true&allowPublicKeyRetrieval=true&nullCatalogMeansCurrent=true"
driver = "com.mysql.cj.jdbc.Driver"
username = "root"
password = "6kF75pjtsauaWCuy"
generate_sink_sql = true
database = "shop_fox_prod"
table = "lazada_order_details"
primary_keys = ["id", "created_at"]
}
}
```
### Running Command
```shell
http://172.31.204.230:25801/hazelcast/rest/maps/submit-job?format=hocon&jobName=LazadaOrderDetails
```
### Error Exception
```log
null
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.