apache / apache/seatunnel

PostgreSQL CDC method syncs data to PolarDB, causing an error. What's the problem?

Open
#9,750 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 17h
Merged PRs (30d)
210

Description

【**Configuration**】:
env {
job.mode = "STREAMING" # CDC必须为流模式
checkpoint.interval = 10000 # 检查点间隔10秒(保障Exactly-once)
}

source {
postgres-cdc {
username = "postgres"
password = "F3456z#"
database-names = ["fsxcd"]
table-names = ["db_1.schema_1.table_1"] # 需同步的表
base-url = "jdbc:postgresql://10.65.21.57:5432/fsxcd?loggerLevel=OFF"
}
}

sink {
jdbc {
url = "jdbc:mysql://10.30.52.17:3306/biz_db"
driver = "com.mysql.cj.jdbc.Driver"
user = "wqs"
password = "fsdgsdfgsdc"
table = "${table_name}" # 动态表名映射
data_save_mode = "UPSERT" # 启用主键去重
primary_keys = ["id"] # 显式指定主键(避免占位符解析失败)
generate_sink_sql = true # 自动建表(依据源表结构)
}
}

【**exception**】:
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'postgres-cdc'.
at org.apache.seatunnel.api.table.factory.FactoryUtil.restoreAndPrepareSource(FactoryUtil.java:168)
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:86)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:376)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:227)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:123)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:191)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:165)
... 2 more
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: Can not find catalog with factoryId [postgres-cdc]
at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.lambda$getCatalogTables$1(CatalogTableUtil.java:141)
at java.base/java.util.Optional.orElseThrow(Optional.java:408)
at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.getCatalogTables(CatalogTableUtil.java:138)
at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.getCatalogTables(CatalogTableUtil.java:98)
at org.apache.seatunnel.connectors.seatunnel.cdc.postgres.source.PostgresIncrementalSourceFactory.lambda$createSource$1(PostgresIncrementalSourceFactory.java:89)
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:180)
at org.apache.seatunnel.api.table.factory.FactoryUtil.restoreAndPrepareSource(FactoryUtil.java:136)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the stack trace at connectors/seatunnel-cdc/postgres/source/PostgresIncrementalSourceFactory.java:89 and follow the catalog lookup for factoryId [postgres-cdc]. Run the supplied configuration in the relevant SeaTunnel environment and verify that source initialization succeeds and the CDC job can proceed without the FactoryException.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.