CDC decoder: translated tuples leak into ReorderBufferChange, causing SIGSEGV in SlabFree during reorder buffer cleanup
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
## Summary
`TranslateChangesIfSchemaChanged()` overwrites `change->data.tp.newtuple` / `oldtuple` with tuples allocated by `heap_form_tuple()` in the current memory context, and never restores the original pointers. PostgreSQL later frees those fields itself in `ReorderBufferFreeChange()`, expecting memory it allocated from the reorder buffer's own tuple context. Freeing a foreign pointer through the slab allocator produces a garbage context pointer and segfaults the walsender.
This is a **pre-existing latent bug**, not a regression from any recent PR. It surfaces intermittently, so it can look like a flaky test.
## Evidence
Observed in `src/test/cdc/t/006_cdc_schema_change_and_move.pl` (PG18), which combines `DROP COLUMN`, a `force_logical` shard move, and a subsequent `UPDATE` — exactly the path that triggers translation. The test aborts after two assertions when the publisher crashes; the visible CI symptom is only a `poll_query_until` catch-up timeout with "connection refused", because the server logs are not uploaded as artifacts.
Backtrace from a local reproduction with a core dump:
```text
SlabFree
ReorderBufferFreeChange
ReorderBufferCleanupTXN
ReorderBufferProcessTXN
xact_decode
LogicalDecodingProcessRecord
XLogSendLogical
WalSndLoop
```
Server log:
```text
LOG: client backend (PID ...) was terminated by signal 11: Segmentation fault
DETAIL: Failed process was running: START_REPLICATION SLOT "cdc_replication_slot" LOGICAL 0/0
(proto_version '4', streaming 'parallel', origin 'any', publication_names '"cdc_publication"')
LOG: shutting down because "restart_after_crash" is off
```
Core inspection identifies the freed object precisely:
- `ReorderBufferChange` with action `REORDER_BUFFER_CHANGE_UPDATE`, XID 797
- the fault occurs on the field at offset `0x38`, confirmed via DWARF to be `data.tp.newtuple`
- the new tuple is still a valid `HeapTupleData` (len 58), i.e. our translated tuple
- its chunk header selects allocator ID 5, from which `SlabFree` derives a bogus context pointer `0x1000000` and faults
So the crash is freeing the **translated** tuple, not the `ReorderBufferChange` container.
## Code path
- `cdc_decoder.c` `GetTupleForTargetSchemaForCdc()` builds a replacement tuple with `heap_form_tuple()`
- `TranslateChangesIfSchemaChanged()` assigns it to `change->data.tp.newtuple` (and `oldtuple` for UPDATE with REPLICA IDENTITY FULL, and for DELETE)
- `TranslateAndPublishRelationForCDC()` calls the pgoutput callback and returns without restoring the original pointers
- PostgreSQL's `ReorderBufferProcessTXN()` calls `AbortCurrentTransaction()` and then `ReorderBufferCleanupTXN()`, which frees both tuple fields
The lifetime mismatch means PostgreSQL frees memory Citus allocated, and Citus's tuples are additionally orphaned if the context is already gone. Intermittency comes from memory reuse, not from concurrency — it does not require a race to fail.
## Fix direction
Restore the original tuple pointers on the `ReorderBufferChange` after publishing, and free the translated tuples ourselves, so each allocator only frees what it owns.
## Notes
- Reproduced with the exact CI-built extension binaries; the previous *passing* CI artifact reproduces it too, and their `.text` sections are byte-identical — confirming this is not introduced by recent packaging changes.
- `.github/actions/save_logs_and_results` does not upload `src/test/cdc/log/`, so CDC TAP server logs are lost in CI. Worth fixing separately to make future crashes diagnosable.
Contributor guide
Research direction
Start with cdc_decoder.c, especially GetTupleForTargetSchemaForCdc(), TranslateChangesIfSchemaChanged(), and TranslateAndPublishRelationForCDC(). Run src/test/cdc/t/006_cdc_schema_change_and_move.pl to reproduce the crash and inspect the ReorderBuffer cleanup path. Done means translated tuples are freed by Citus, original pointers remain available to PostgreSQL cleanup, and the CDC test completes without the walsender crashing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, postgresql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100