A long running PostgresTaskContext.refreshSchema causes the replication stream to fail [DBZ-932]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-932](https://issues.redhat.com/browse/DBZ-932)
If `PostgresTaskContext.refreshSchema` takes longer than `wal_sender_timeout` to run the replication stream errors on the first call to `read`.
{noformat}
org.apache.kafka.connect.errors.ConnectException: An exception ocurred in the change event producer. This connector will be stopped.
at io.debezium.connector.base.ChangeEventQueue.throwProducerFailureIfPresent(ChangeEventQueue.java:168)
at io.debezium.connector.base.ChangeEventQueue.poll(ChangeEventQueue.java:149)
at io.debezium.connector.postgresql.PostgresConnectorTask.poll(PostgresConnectorTask.java:146)
at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:244)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:220)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: Database connection failed when reading from copy
at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:964)
at org.postgresql.core.v3.CopyDualImpl.readFromCopy(CopyDualImpl.java:41)
at org.postgresql.core.v3.replication.V3PGReplicationStream.receiveNextData(V3PGReplicationStream.java:145)
at org.postgresql.core.v3.replication.V3PGReplicationStream.readInternal(V3PGReplicationStream.java:114)
at org.postgresql.core.v3.replication.V3PGReplicationStream.read(V3PGReplicationStream.java:60)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:198)
at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:128)
at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$1(RecordsStreamProducer.java:114)
... 5 more
Caused by: java.io.EOFException
at org.postgresql.core.PGStream.receiveChar(PGStream.java:284)
at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1006)
at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:962)
... 12 more
{noformat}
We run a tenant per schema which each schema having ~50 tables. Refreshing the schema takes over a minute.
We have worked around this by using the fact that we set `schema.whitelist` to a single schema and passing that through to `connection.readSchema`.
{noformat}
this.databaseCatalog = config.databaseName();
String schemaWhitelist = config.schemaWhitelist();
if (schemaWhitelist != null && !schemaWhitelist.isEmpty()) {
// not valid if there is regex in this string or it has commas, postgresql puts this string into a LIKE
this.schemaNamePattern = schemaWhitelist;
} else {
this.schemaNamePattern = null;
}
...
connection.readSchema(tables(), this.databaseCatalog, this.schemaNamePattern, filters.tableNameFilter(), null, true);
{noformat}
Contributor guide
Research direction
Start with PostgresTaskContext.refreshSchema and trace how it interacts with PostgresReplicationConnection.read and RecordsStreamProducer.streamChanges. Reproduce with a schema refresh longer than PostgreSQL wal_sender_timeout, then verify the replication stream remains usable on its first read after refresh completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgresql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100