PostgreSQL CDC method syncs data to PolarDB, causing an error. What's the problem?
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 210
Description
【**Configuration**】:
env {
job.mode = "STREAMING" # CDC必须为流模式
checkpoint.interval = 10000 # 检查点间隔10秒(保障Exactly-once)
}
source {
postgres-cdc {
username = "postgres"
password = "F3456z#"
database-names = ["fsxcd"]
table-names = ["db_1.schema_1.table_1"] # 需同步的表
base-url = "jdbc:postgresql://10.65.21.57:5432/fsxcd?loggerLevel=OFF"
}
}
sink {
jdbc {
url = "jdbc:mysql://10.30.52.17:3306/biz_db"
driver = "com.mysql.cj.jdbc.Driver"
user = "wqs"
password = "fsdgsdfgsdc"
table = "${table_name}" # 动态表名映射
data_save_mode = "UPSERT" # 启用主键去重
primary_keys = ["id"] # 显式指定主键(避免占位符解析失败)
generate_sink_sql = true # 自动建表(依据源表结构)
}
}
【**exception**】:
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'postgres-cdc'.
at org.apache.seatunnel.api.table.factory.FactoryUtil.restoreAndPrepareSource(FactoryUtil.java:168)
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:86)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:376)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:227)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:123)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:191)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:165)
... 2 more
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: Can not find catalog with factoryId [postgres-cdc]
at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.lambda$getCatalogTables$1(CatalogTableUtil.java:141)
at java.base/java.util.Optional.orElseThrow(Optional.java:408)
at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.getCatalogTables(CatalogTableUtil.java:138)
at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.getCatalogTables(CatalogTableUtil.java:98)
at org.apache.seatunnel.connectors.seatunnel.cdc.postgres.source.PostgresIncrementalSourceFactory.lambda$createSource$1(PostgresIncrementalSourceFactory.java:89)
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:180)
at org.apache.seatunnel.api.table.factory.FactoryUtil.restoreAndPrepareSource(FactoryUtil.java:136)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the stack trace at connectors/seatunnel-cdc/postgres/source/PostgresIncrementalSourceFactory.java:89 and follow the catalog lookup for factoryId [postgres-cdc]. Run the supplied configuration in the relevant SeaTunnel environment and verify that source initialization succeeds and the CDC job can proceed without the FactoryException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100