[source-postgresql] CDC sync is only allowing WAL to be emptied every two syncs
- Vorherrschende Sprache
- Python
- Sterne
- 22.1k
- Forks
- 5.3k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
### Connector Name
source-postgresql
### Connector Version
3.6.15
### What step the error happened?
During the sync
### Relevant information
We are using the `3.6.15` version of the connector to run CDC jobs. the reason why we are using this connector version, and not a more recent one is because of the serious CDC issue identified in this github issue (https://github.com/airbytehq/airbyte/issues/55846).
During the sync, events are correctly collected but the WAL is not emptied on PostgreSQL side, when it should given all registered consumers of a PostgreSQL publication have already consumed the data. Below the exact flow of events.
1. During the sync that ingests data, we get the below logs in a repeated fashion indicating that there are no more events to consume
```shell
2025-12-12 09:04:47 source INFO pool-2-thread-1 i.d.c.c.BaseSourceTask(logStatistics):349 15 records sent during previous 00:02:32.367, last recorded offset of {server=multicurrency-preprod-agicap} partition is {lsn_proc=2211942359720, messageType=COMMIT, lsn_commit=2211942359720, lsn=2211942359720, txId=9183903, ts_usec=1765475214558055}
2025-12-12 09:04:47 source INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=31, polls=0
...
2025-12-12 09:05:38 source INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=36, polls=0
2025-12-12 09:05:38 source INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.521946024S after its previous call which was also logged.
2025-12-12 09:05:38 source INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
```
2. Once this sync is done, we see on the PostgreSQL instance that the WAL has not been emptied.
3. We then launch another sync, this time zero rows are ingested, we see a log similar to the one below.
```shell
2025-12-12 09:41:00 source INFO pool-2-thread-1 i.d.c.p.PostgresStreamingChangeEventSource(commitOffset):427 Received offset commit request on 'LSN{203/209E2A8}', but ignoring it. LSN flushing is not allowed yet
```
4. Once this second sync is done, the WAL is emptied. As a consequence, it seems that the WAL is only emptied once every two syncs.
This behavior seems arbitrary and if there is a technical explanation behind it, it should be described in the documentation.
### Contribute
- [ ] Yes, I want to contribute
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.