Postgres connector consumes all connections when restarting [DBZ-9537]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-9537](https://issues.redhat.com/browse/DBZ-9537)
In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
h1. Bug report
When replication slot is lost, connector attempts to restart. If errors.max.retries config isn't set connector will always restart. Debezium postgres task has two postgres connections in normal condition. However when connector's task is restarted due to replication slot's loss, the connections don't close. They stay in idle state (you can see it in pg_stat_activity). When task restart's it fails again due to lost slot and leaves 2 new connections in idle state. It continues further and finally, debezium connector consumes all available connections. After that task goes to failed status, but all connections stays in idle status and aren't closed even if idle_session_timeout is configured.
h2. What Debezium connector do you use and what version?
2.6.2-Final
h2. What is the connector configuration?
h2. What is the captured database version and mode of deployment?
Postgres 16
h2. What behavior do you expect?
Connections of previous task are closed when it restarts
h2. What behavior do you see?
Connections stay in idle status
h2. Do you see the same behaviour using the latest released Debezium version?
Didn't check
h2. Do you have the connector logs, ideally from start till finish?
(You might be asked later to provide [DEBUG/TRACE|https://debezium.io/documentation/reference/stable/operations/logging.html] level log)
I'll provide it if required.
h2. How to reproduce the issue using our [tutorial|https://github.com/debezium/debezium-examples/tree/main/tutorial] deployment?
setup low max slot wal keep size for postgres, disable heartbeat table, execute queries on table of same db, which isn't tracked by debezium. Wait till replication slot gets lost. Observe consuming connections.
Contributor guide
Research direction
Start by reproducing the restart scenario from the debezium-examples/tutorial deployment with the described low WAL keep size and disabled heartbeat table, then observe pg_stat_activity during replication-slot loss. Done means previous-task connections no longer remain idle after restarts and the connector does not exhaust available PostgreSQL connections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100