apache / apache/seatunnel

[Bug] [Postgres-CDC] Failed to synchronize partitioned tables to MySQL using Postgres-CDC

Open
#10,593 10 comments 0 reactions 1 assignee Claimed by @misi1987107 View on GitHub
bug good first issue help wanted
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

我通过/hazelcast/rest/maps/submit-job进行提交任务,现在测试阶段打算把一个Postgres的分区表(pg_partman自动分区)的数据以流处理的方式同步到MySQL中,hocon配置如下
```hocon
env {
job.mode = "STREAMING"
execution.parallelism = 1
checkpoint.interval = 10000
}

source {
Postgres-CDC {
url = "jdbc:postgresql://172.31.204.230:5432/datahub"
username = "datahub"
password = "1111"
database-names = ["datahub"]
schema-names = ["original"]
table-names = ["datahub.ods.lazada_order_details"]

startup.mode = "latest"
slot.name = "seatunnel_slot"
decoding.plugin.name = "pgoutput"
}
}

transform {

}

sink {
Jdbc {
url = "jdbc:mysql://172.31.204.227:3306/shop_fox_prod?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true&allowPublicKeyRetrieval=true&nullCatalogMeansCurrent=true"
driver = "com.mysql.cj.jdbc.Driver"
username = "root"
password = "111"

generate_sink_sql = true
database = "shop_fox_prod"
table = "lazada_order_details"

primary_keys = ["id", "created_at"]
}
}
```
在web界面能看到任务长期处于RUNNING状态,但是source的TableSourceReceivedCount字段始终为0,sink的TableSinkWriteCount字段也为0,目前使用的是apache/seatunnel:2.3.13的docker镜像,默认使用Zeta引擎,

然后观察日志发现有注意到:SELECT reltuples FROM pg_class r WHERE relkind = 'r' AND relname = 'lazada_order_details'; 的SQL,是因为现在的Postgres-CDC只能同步普通表吗?

Image

### SeaTunnel Version

2.3.13

### SeaTunnel Config

```conf
env {
job.mode = "STREAMING"
execution.parallelism = 1
checkpoint.interval = 10000
}

source {
Postgres-CDC {
url = "jdbc:postgresql://172.31.204.230:5432/datahub"
username = "datahub"
password = "TeKvkjtFtBceyCF6"
database-names = ["datahub"]
schema-names = ["original"]
table-names = ["datahub.original.lazada_order_details"]

startup.mode = "latest"
slot.name = "seatunnel_slot"
decoding.plugin.name = "pgoutput"
}
}

transform {

}

sink {
Jdbc {
url = "jdbc:mysql://172.31.204.227:3306/shop_fox_prod?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true&allowPublicKeyRetrieval=true&nullCatalogMeansCurrent=true"
driver = "com.mysql.cj.jdbc.Driver"
username = "root"
password = "6kF75pjtsauaWCuy"

generate_sink_sql = true
database = "shop_fox_prod"
table = "lazada_order_details"

primary_keys = ["id", "created_at"]
}
}
```

### Running Command

```shell
http://172.31.204.230:25801/hazelcast/rest/maps/submit-job?format=hocon&jobName=LazadaOrderDetails
```

### Error Exception

```log
null
```

### Zeta or Flink or Spark Version

_No response_

### Java or Scala Version

_No response_

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.