citusdata / citusdata/citus

citus_decoders/pgoutput.so segfaults (signal 11) when decoding WAL containing ALTER TABLE ADD/DROP COLUMN

Open
#8,562 2 comments 0 reactions 1 assignee Claimed by @emelsimsek View on GitHub
needs-investigation
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Environment:

Citus 14.0, PostgreSQL 18
wal_level = logical
citus.enable_change_data_capture = true

Steps to reproduce:

Create a distributed table with some data
Create a publication: CREATE PUBLICATION test_pub FOR TABLE my_table;
Create a logical replication slot: SELECT pg_create_logical_replication_slot('test_slot', 'pgoutput', false);
Set up a subscriber on a standalone PostgreSQL instance using that publication and slot — CDC works fine
Run ALTER TABLE my_table ADD COLUMN test_col boolean; on the coordinator
Run ALTER TABLE my_table DROP COLUMN test_col; on the coordinator
The CDC decoder segfaults on every worker that has shards of the altered table, crash-looping the entire postgres process every ~5 seconds

Behavior:
The Citus CDC decoder (citus_decoders/pgoutput.so) crashes when it encounters WAL entries written with a different column count than the current catalog state. The segfault takes down the entire postgres process on the worker — not just the walsender — disrupting all queries, not just CDC.
Once triggered, the crash loop is unrecoverable without disabling CDC entirely. Dropping and recreating slots does not help — the decoder scans back to find a consistent decoding point and hits the corrupted WAL region again. pg_replication_slot_advance also does not help. The only fix is disabling citus.enable_change_data_capture, removing all slots, and waiting for the affected WAL to age out.

Expected:
Well, it shouldn't segfault. It's really annoying issue, because for the rest if it works, CITUS CDC is a breeze to work with truly. But, if it segfautls bc of 1 DDL change, then it's completely unusable.

Note: I would like a work-around if there is any, otherwise I'll wait for a fix, but I am kinda out of solutions. I did attach a strack trace.

GDB stacktrace:
```
[New LWP 3399]

warning: Section `.reg-xstate/3399' in core file too small.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: 18/main: walsender REDACTED_ROLE DATABASE_NAME_REDACTED IP_REDACTED(36102) STA'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Section `.reg-xstate/3399' in core file too small.
#0 fill_val (att=att@entry=0x7fe7215ef488, bit=, bitmask=bitmask@entry=0x7fffaf2373ac, dataP=dataP@entry=0x7fffaf237398, infomask=infomask@entry=0x563efa55fce4, datum=0, isnull=) at ./build/../src/backend/access/common/heaptuple.c:327
327 ./build/../src/backend/access/common/heaptuple.c: No such file or directory.
#0 fill_val (att=att@entry=0x7fe7215ef488, bit=, bitmask=bitmask@entry=0x7fffaf2373ac, dataP=dataP@entry=0x7fffaf237398, infomask=infomask@entry=0x563efa55fce4, datum=0, isnull=) at ./build/../src/backend/access/common/heaptuple.c:327
val = 0x0
data_length =
data = 0x563efa55fcf4 ""
#1 0x0000563ef804f5bb in heap_fill_tuple (tupleDesc=tupleDesc@entry=0x7fe7215ef460, values=values@entry=0x563efa55fd10, isnull=isnull@entry=0x563efa55fd98, data=, data_size=data_size@entry=296, infomask=infomask@entry=0x563efa55fce4, bit=) at ./build/../src/backend/access/common/heaptuple.c:433
attr = 0x7fe7215ef488
bitP = 0x563efa55fce7 "\003"
bitmask = 2
i =
numberOfAttributes = 13
#2 0x0000563ef804fe70 in heap_form_tuple (tupleDescriptor=0x7fe7215ef460, values=, isnull=) at ./build/../src/backend/access/common/heaptuple.c:1187
tuple = 0x563efa55fcb8
td = 0x563efa55fcd0
len =
data_len =
hoff =
hasnull =
numberOfAttributes = 13
i =
__func__ = "heap_form_tuple"
#3 0x00007fef5df48840 in ?? () from /usr/lib/postgresql/18/lib/citus_decoders/pgoutput.so
No symbol table info available.
#4 0x0000563ef83567e2 in change_cb_wrapper (cache=, txn=, relation=, change=) at ./build/../src/backend/replication/logical/logical.c:1120
ctx =
state = {ctx = 0x563efa4e3248, callback_name = 0x563ef86d07c5 "change", report_location = 61961555667384}
errcallback = {previous = 0x0, callback = 0x563ef8357630 , arg = 0x7fffaf2374f0}
#5 0x0000563ef8361a03 in ReorderBufferApplyChange (streaming=false, change=0x563efa5d5ca0, relation=0x7fe7215ee508, txn=0x563efa5cb7d0, rb=0x563efa5ad688) at ./build/../src/backend/replication/logical/reorderbuffer.c:2078
No locals.
#6 ReorderBufferProcessTXN (rb=, txn=, commit_lsn=, snapshot_now=, command_id=command_id@entry=0, streaming=streaming@entry=false) at ./build/../src/backend/replication/logical/reorderbuffer.c:2383
relation =
reloid =
change = 0x563efa5d5ca0
changes_count =
_save_exception_stack =
_save_context_stack =
_local_sigjmp_buf = {{__jmpbuf = {0, -577971927189847483, 0, 0, 94828487389768, 140736131726416, -577971927114350011, -6595504393366437307}, __mask_was_saved = 0, __saved_mask = {__val = {0, 0, 1, 0, 6887684528167130792, 140736131725824, 9058412190089515264, 140736131725968, 94828488218248, 61961555678864, 831387897450244, 0, 0, 140736131725952, 94828452261188, 0}}}}
_do_rethrow =
using_subtxn =
ccxt =
iterstate = 0x563efa5ad890
prev_lsn = 61961555667384
specinsert = 0x0
stream_started = false
curtxn = 0x0
__func__ = "ReorderBufferProcessTXN"
change_done =
#7 0x0000563ef83622aa in ReorderBufferReplay (txn=, rb=, commit_lsn=, end_lsn=, commit_time=, origin_id=, origin_lsn=, xid=) at ./build/../src/backend/replication/logical/reorderbuffer.c:2864
snapshot_now =
command_id = 0
#8 0x0000563ef8362d74 in ReorderBufferCommit (rb=, xid=, commit_lsn=9058412190089515264, end_lsn=, commit_time=, origin_id=, origin_lsn=) at ./build/../src/backend/replication/logical/reorderbuffer.c:2888
txn =
#9 0x0000563ef8353b62 in DecodeCommit (two_phase=false, xid=1603664021, parsed=0x7fffaf237ac0, buf=0x7fffaf237c50, ctx=0x563efa4e3248) at ./build/../src/backend/replication/logical/decode.c:734
origin_lsn =
commit_time =
origin_id =
i =
origin_lsn =
commit_time =
origin_id =
i =
#10 xact_decode (ctx=0x563efa4e3248, buf=0x7fffaf237c50) at ./build/../src/backend/replication/logical/decode.c:242
```

If this could be fixed, if there is a work-around I would like that.

I have tried to drop slots, recreate the pubs. Synchronize the schemas all. But, none of that is working, because as soon as I turn on the subscription on the subscriber, Citus CDC decoder starts segfaulting.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.