Postgresql connector start is slow on multitenancy cases [DBZ-4840]
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from DBZ-4840
The postgresql connector is really slow to start up on my infrastructure (takes 17h).
For context, I'm using a postgresql DB with multitenancy (1 tenant = 1 schema), 4000 schemas and ~15 tables per schemas monitored.
Digging up, I found out it passed the most of that time [on this instruction|https://github.com/debezium/debezium/blob/6e581599c9e4261e446670332567dc1e6cd922c6/debezium-core/src/main/java/io/debezium/jdbc/JdbcConnection.java#L1201], a loop on every table to be snapshotted. On my connector, it was going at a speed of ~2 tables / seconds.
This behavior was introduced with DBZ-2575 (which does not improve performance when the number of tables to query is high), and can be set back to precedent using the parameter database.internal.snapshot.scan.all.columns.force: true (I spent way too long to find the needed prefix for this parameter). With this parameter set, this step now takes 5m on my infra (vs 17h).
Since this is an "internal" switch, I understand it isn't documented / might be changed at any moment. I'd suggest making this switch official and documented (e.g. on https://debezium.io/documentation/faq/, where there's already some good tips on handling large table numbers), and maybe send a log "info" if the number of tables to query is big (>600) with this parameter disabled as it can then takes a long time.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at debezium-core/src/main/java/io/debezium/jdbc/JdbcConnection.java around line 1201 and trace how database.internal.snapshot.scan.all.columns.force is handled during table snapshotting. Review the Debezium FAQ section on large table counts; done means the switch is officially documented and the proposed large-table startup warning is addressed or explicitly scoped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgresql
- Domain
- databases, documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100