citus_decoders/pgoutput.so segfaults (signal 11) when decoding WAL containing ALTER TABLE ADD/DROP COLUMN
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Environment:
Citus 14.0, PostgreSQL 18
wal_level = logical
citus.enable_change_data_capture = true
Steps to reproduce:
Create a distributed table with some data
Create a publication: CREATE PUBLICATION test_pub FOR TABLE my_table;
Create a logical replication slot: SELECT pg_create_logical_replication_slot('test_slot', 'pgoutput', false);
Set up a subscriber on a standalone PostgreSQL instance using that publication and slot — CDC works fine
Run ALTER TABLE my_table ADD COLUMN test_col boolean; on the coordinator
Run ALTER TABLE my_table DROP COLUMN test_col; on the coordinator
The CDC decoder segfaults on every worker that has shards of the altered table, crash-looping the entire postgres process every ~5 seconds
Behavior:
The Citus CDC decoder (citus_decoders/pgoutput.so) crashes when it encounters WAL entries written with a different column count than the current catalog state. The segfault takes down the entire postgres process on the worker — not just the walsender — disrupting all queries, not just CDC.
Once triggered, the crash loop is unrecoverable without disabling CDC entirely. Dropping and recreating slots does not help — the decoder scans back to find a consistent decoding point and hits the corrupted WAL region again. pg_replication_slot_advance also does not help. The only fix is disabling citus.enable_change_data_capture, removing all slots, and waiting for the affected WAL to age out.
Expected:
Well, it shouldn't segfault. It's really annoying issue, because for the rest if it works, CITUS CDC is a breeze to work with truly. But, if it segfautls bc of 1 DDL change, then it's completely unusable.
Note: I would like a work-around if there is any, otherwise I'll wait for a fix, but I am kinda out of solutions. I did attach a strack trace.
GDB stacktrace:
```
[New LWP 3399]
warning: Section `.reg-xstate/3399' in core file too small.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: 18/main: walsender REDACTED_ROLE DATABASE_NAME_REDACTED IP_REDACTED(36102) STA'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Section `.reg-xstate/3399' in core file too small.
#0 fill_val (att=att@entry=0x7fe7215ef488, bit=, bitmask=bitmask@entry=0x7fffaf2373ac, dataP=dataP@entry=0x7fffaf237398, infomask=infomask@entry=0x563efa55fce4, datum=0, isnull=) at ./build/../src/backend/access/common/heaptuple.c:327
327 ./build/../src/backend/access/common/heaptuple.c: No such file or directory.
#0 fill_val (att=att@entry=0x7fe7215ef488, bit=, bitmask=bitmask@entry=0x7fffaf2373ac, dataP=dataP@entry=0x7fffaf237398, infomask=infomask@entry=0x563efa55fce4, datum=0, isnull=) at ./build/../src/backend/access/common/heaptuple.c:327
val = 0x0
data_length =
data = 0x563efa55fcf4 ""
#1 0x0000563ef804f5bb in heap_fill_tuple (tupleDesc=tupleDesc@entry=0x7fe7215ef460, values=values@entry=0x563efa55fd10, isnull=isnull@entry=0x563efa55fd98, data=, data_size=data_size@entry=296, infomask=infomask@entry=0x563efa55fce4, bit=) at ./build/../src/backend/access/common/heaptuple.c:433
attr = 0x7fe7215ef488
bitP = 0x563efa55fce7 "\003"
bitmask = 2
i =
numberOfAttributes = 13
#2 0x0000563ef804fe70 in heap_form_tuple (tupleDescriptor=0x7fe7215ef460, values=, isnull=) at ./build/../src/backend/access/common/heaptuple.c:1187
tuple = 0x563efa55fcb8
td = 0x563efa55fcd0
len =
data_len =
hoff =
hasnull =
numberOfAttributes = 13
i =
__func__ = "heap_form_tuple"
#3 0x00007fef5df48840 in ?? () from /usr/lib/postgresql/18/lib/citus_decoders/pgoutput.so
No symbol table info available.
#4 0x0000563ef83567e2 in change_cb_wrapper (cache=, txn=, relation=, change=) at ./build/../src/backend/replication/logical/logical.c:1120
ctx =
state = {ctx = 0x563efa4e3248, callback_name = 0x563ef86d07c5 "change", report_location = 61961555667384}
errcallback = {previous = 0x0, callback = 0x563ef8357630 , arg = 0x7fffaf2374f0}
#5 0x0000563ef8361a03 in ReorderBufferApplyChange (streaming=false, change=0x563efa5d5ca0, relation=0x7fe7215ee508, txn=0x563efa5cb7d0, rb=0x563efa5ad688) at ./build/../src/backend/replication/logical/reorderbuffer.c:2078
No locals.
#6 ReorderBufferProcessTXN (rb=, txn=, commit_lsn=, snapshot_now=, command_id=command_id@entry=0, streaming=streaming@entry=false) at ./build/../src/backend/replication/logical/reorderbuffer.c:2383
relation =
reloid =
change = 0x563efa5d5ca0
changes_count =
_save_exception_stack =
_save_context_stack =
_local_sigjmp_buf = {{__jmpbuf = {0, -577971927189847483, 0, 0, 94828487389768, 140736131726416, -577971927114350011, -6595504393366437307}, __mask_was_saved = 0, __saved_mask = {__val = {0, 0, 1, 0, 6887684528167130792, 140736131725824, 9058412190089515264, 140736131725968, 94828488218248, 61961555678864, 831387897450244, 0, 0, 140736131725952, 94828452261188, 0}}}}
_do_rethrow =
using_subtxn =
ccxt =
iterstate = 0x563efa5ad890
prev_lsn = 61961555667384
specinsert = 0x0
stream_started = false
curtxn = 0x0
__func__ = "ReorderBufferProcessTXN"
change_done =
#7 0x0000563ef83622aa in ReorderBufferReplay (txn=, rb=, commit_lsn=, end_lsn=, commit_time=, origin_id=, origin_lsn=, xid=) at ./build/../src/backend/replication/logical/reorderbuffer.c:2864
snapshot_now =
command_id = 0
#8 0x0000563ef8362d74 in ReorderBufferCommit (rb=, xid=, commit_lsn=9058412190089515264, end_lsn=, commit_time=, origin_id=, origin_lsn=) at ./build/../src/backend/replication/logical/reorderbuffer.c:2888
txn =
#9 0x0000563ef8353b62 in DecodeCommit (two_phase=false, xid=1603664021, parsed=0x7fffaf237ac0, buf=0x7fffaf237c50, ctx=0x563efa4e3248) at ./build/../src/backend/replication/logical/decode.c:734
origin_lsn =
commit_time =
origin_id =
i =
origin_lsn =
commit_time =
origin_id =
i =
#10 xact_decode (ctx=0x563efa4e3248, buf=0x7fffaf237c50) at ./build/../src/backend/replication/logical/decode.c:242
```
If this could be fixed, if there is a work-around I would like that.
I have tried to drop slots, recreate the pubs. Synchronize the schemas all. But, none of that is working, because as soon as I turn on the subscription on the subscriber, Citus CDC decoder starts segfaulting.
Contributor guide
Assessment
This issue has not been assessed yet.