ClickHouse / ClickHouse/ClickHouse
MaterializedPostgreSQL silently loses acknowledged rows after power loss: the replication slot is advanced before the target part is durable
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Describe what's wrong
A `MaterializedPostgreSQL` database applies a batch of decoded changes to its ClickHouse-side nested table with a plain (un-fsynced) `INSERT` and then, in the same step, durably advances the PostgreSQL logical-replication slot (`pg_replication_slot_advance`). On restart it resumes **only** from the slot's `confirmed_flush_lsn` — it keeps no durable, independent record of how far it has actually persisted. So if ClickHouse loses power after the slot has advanced but before the target part is durable, the acknowledged rows are gone from ClickHouse while PostgreSQL has already advanced (and is free to recycle the WAL) past them. They are **never re-sent**: the replica silently and permanently diverges from its source.
This is the same "a durable cursor certifies non-durable data" class as the accepted `S3Queue` (#111537) and `FileLog` (#111722) reports, but on a third, distinct subsystem — logical replication from an external PostgreSQL server — and with strictly worse recoverability, because advancing the slot irreversibly recycles the source WAL (the source cannot be asked to resend).
| | rows acked by ClickHouse | in ClickHouse after power loss | re-sent by PostgreSQL |
|---|---|---|---|
| batch of 300 rows | yes (applied + slot advanced) | **0** | never (slot advanced past them) |
Deterministic: 3/3 on `master` (26.7.1.1380) and reproduced on 26.3.2.3; a graceful restart of the same batch loses nothing.
### Does it reproduce on the most recent release?
Yes — reproduced on `master` (26.7.1.1380) and on 26.3.2.3. The code path is unchanged for years, so this is longstanding rather than a recent regression.
### How to reproduce
1. Start PostgreSQL with `wal_level=logical`, create `t (id int primary key, v int)`, insert an initial 100 rows.
2. In ClickHouse:
```sql
CREATE DATABASE mpg ENGINE = MaterializedPostgreSQL('pg:5432', 'postgres', 'postgres', 'pg')
SETTINGS materialized_postgresql_tables_list = 't';
```
Wait for the initial snapshot (`mpg.t` has 100 rows), then `sync` so everything up to here is durable.
3. Insert a batch into PostgreSQL: `INSERT INTO t SELECT g, g*7 FROM generate_series(101,400) g;` Wait until `mpg.t` shows 400 rows **and** `confirmed_flush_lsn` in `pg_replication_slots` has advanced past the batch.
4. Simulate power loss on the ClickHouse data directory (drop un-fsynced writes) and kill the server — e.g. a `dm-flakey` `drop_writes` cut, then remount.
5. Restart ClickHouse. Insert one fresh marker row in PostgreSQL (`INSERT INTO t VALUES (999999, 7);`).
Observed after restart: `mpg.t` contains the 100 baseline rows and the marker row (id 999999), but **none** of the 300 batch rows (ids 101–400). The marker's arrival proves replication resumed live from the advanced slot, so the batch is permanently lost — PostgreSQL will not resend it.
The finding comes from a crash-durability test harness. Trust guards used: a capability probe (the `MaterializedPostgreSQL` batch must actually stream into the nested table on this binary); a vacuity gate (the batch must be applied in ClickHouse **and** the slot must have advanced past it before the cut — otherwise a missing row would just be re-delivered on resume); a graceful control (the same batch with a full `sync` before a clean restart loses nothing — so the loss is power-loss-specific, not a generic restart bug); and 3× repeats for determinism.
### Expected behavior
An acknowledged CDC replica should not lose acknowledged rows across a power loss. The PostgreSQL slot should not be advanced past rows whose ClickHouse-side part is not yet durable — e.g. fsync the target part before `advanceLSN`, or persist a ClickHouse-side applied-LSN durably and resume from `min(slot_lsn, durable_applied_lsn)` so a lost batch is re-streamed instead of skipped.
### Error message and/or stacktrace
None — silent. No exception is raised; the replica simply and permanently has fewer rows than the source.
### Additional context
Code walk (master 26.7.1.1380)
`MaterializedPostgreSQLConsumer::syncTables` applies the batch and then advances the slot in the same call:
```cpp
// src/Storages/PostgreSQL/MaterializedPostgreSQLConsumer.cpp
CompletedPipelineExecutor executor(io.pipeline); // ~line 718
executor.execute(); // ~line 719 plain INSERT, async_insert=false, no fsync
...
updateLsn(); // ~line 738
void MaterializedPostgreSQLConsumer::updateLsn()
{
auto tx = std::make_shared(connection->getRef());
current_lsn = advanceLSN(tx); // ~line 746
tables_to_sync.clear();
tx->commit(); // ~line 748 durable on the PG server
}
String MaterializedPostgreSQLConsumer::advanceLSN(...)
{
// SELECT end_lsn FROM pg_replication_slot_advance('', '') ~line 758
...
}
```
On restart the resume position is the slot's `confirmed_flush_lsn`, read back from PostgreSQL — there is no durable ClickHouse-side applied LSN:
```cpp
// src/Storages/PostgreSQL/PostgreSQLReplicationHandler.cpp
/// 2. if replication slot already exists, start_lsn is read from pg_replication_slots as
/// `confirmed_flush_lsn` - the address (LSN) up to which the logical slot's
/// consumer has confirmed receiving data. // ~line 319
...
// SELECT active, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots ... // ~line 667
start_lsn = result[0][2].as(); // ~line 674
```
The target `INSERT` uses the default `fsync_after_insert = 0`, so after `execute()` returns the part is in the OS page cache but not necessarily on stable storage; `advanceLSN` then makes the slot advance durable. A power loss in that window drops the part and leaves the slot advanced past it.
Contributor guide
Assessment
This issue has not been assessed yet.