Backend process crashed when executing 2pc transaction in citus

Open

#3,228 opened on Nov 26, 2019 4 comments 0 reactions 0 assignees View on GitHub

Dominant language: C
Stars: 12.8k
Forks: 794
Avg merge: 2d 14h
Merged PRs (30d): 31

Description

1. phenomenon
Backend process crashed when executing 2pc transaction in citus.

- coordinator

2019-11-24 11:08:09.914 CST 35791 10.246.66.182(6881) lobausr lobadbw2 PostgreSQL JDBC Driver WARNING: server conn crashed?
2019-11-24 11:08:09.914 CST 35791 10.246.66.182(6881) lobausr lobadbw2 PostgreSQL JDBC Driver CONTEXT: while executing command on 10.230.27.117:6432
2019-11-24 11:08:09.914 CST 35791 10.246.66.182(6881) lobausr lobadbw2 PostgreSQL JDBC Driver WARNING: failed to commit transaction on 10.230.27.117:6432
2019-11-24 11:08:09.914 CST 35791 10.246.66.182(6881) lobausr lobadbw2 PostgreSQL JDBC Driver WARNING: failed to roll back prepared transaction 'citus_0_35791_4207001212_1287199'
2019-11-24 11:08:09.914 CST 35791 10.246.66.182(6881) lobausr lobadbw2 PostgreSQL JDBC Driver HINT: Run "COMMIT PREPARED 'citus_0_35791_4207001212_1287199'" on 10.230.27.117:6432
2019-11-24 11:08:09.914 CST 35791 10.246.66.182(6881) lobausr lobadbw2 PostgreSQL JDBC Driver WARNING: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2019-11-24 11:08:09.914 CST 35791 10.246.66.182(6881) lobausr lobadbw2 PostgreSQL JDBC Driver CONTEXT: while executing command on 10.230.27.117:6432
2019-11-24 11:08:09.914 CST 35791 10.246.66.182(6881) lobausr lobadbw2 PostgreSQL JDBC Driver LOG: duration: 17123.210 ms execute S_1: COMMIT

- worker

2019-11-24 11:08:09.854 CST 14668 LOG: server process (PID 14714) was terminated by signal 6: Aborted
2019-11-24 11:08:09.854 CST 14668 DETAIL: Failed process was running: COMMIT PREPARED 'citus_0_35791_4207001212_1287199'
2019-11-24 11:08:09.854 CST 14668 LOG: terminating any other active server processes

2. Occurrence condition
Distributed transaction in business SQL

- PostgreSQL：10.7
- citus：7.4.1
- OS：RHEL6.3

3. Investigation
3.1 PG log
- worker

*** glibc detected *** postgres: lobausr lobadbw2
======= Backtrace: =========
/lib64/libc.so.6[0x369e275f4e]
/lib64/libc.so.6[0x369e278cf0]
postgres: lobausr lobadbw2 127.0.0.1(39349)
postgres: lobausr lobadbw2 127.0.0.1(39349)
postgres: lobausr lobadbw2 127.0.0.1(39349)
postgres: lobausr lobadbw2 127.0.0.1(39349)
/usr/pgsql-10/lib/citus.so(multi_ProcessUtilit
/usr/pgsql-10/lib/pg_stat_statements.so(+0x417
postgres: lobausr lobadbw2 127.0.0.1(39349)
postgres: lobausr lobadbw2 127.0.0.1(39349)
postgres: lobausr lobadbw2 127.0.0.1(39349)
postgres: lobausr lobadbw2 127.0.0.1(39349)
postgres: lobausr lobadbw2 127.0.0.1(39349)
postgres: lobausr lobadbw2 127.0.0.1(39349)
postgres: lobausr lobadbw2 127.0.0.1(39349)
/lib64/libc.so.6(__libc_start_main+0xfd)[0x369
postgres: lobausr lobadbw2 127.0.0.1(39349)
======= Memory map: ========
00400000-00a99000 r-xp 00000000 fd:08 266747
00c99000-00ca7000 rw-p 00699000 fd:08 266747
00ca7000-00d10000 rw-p 00000000 00:00 0
0190c000-019b3000 rw-p 00000000 00:00 0
019b3000-01c5d000 rw-p 00000000 00:00 0
369de00000-369de20000 r-xp 00000000 fd:01 3765
369e01f000-369e020000 r--p 0001f000 fd:01 3765
369e020000-369e021000 rw-p 00020000 fd:01 3765
369e021000-369e022000 rw-p 00000000 00:00 0
369e200000-369e38a000 r-xp 00000000 fd:01 3766
369e38a000-369e58a000 ---p 0018a000 fd:01 3766
369e58a000-369e58e000 r--p 0018a000 fd:01 3766
369e58e000-369e58f000 rw-p 0018e000 fd:01 3766
369e58f000-369e594000 rw-p 00000000 00:00 0
369e600000-369e602000 r-xp 00000000 fd:01 5167
369e602000-369e802000 ---p 00002000 fd:01 5167
369e802000-369e803000 r--p 00002000 fd:01 5167
369e803000-369e804000 rw-p 00003000 fd:01 5167
369ea00000-369ea17000 r-xp 00000000 fd:01 5174
369ea17000-369ec17000 ---p 00017000 fd:01 5174
369ec17000-369ec18000 r--p 00017000 fd:01 5174
369ec18000-369ec19000 rw-p 00018000 fd:01 5174
369ec19000-369ec1d000 rw-p 00000000 00:00 0
369ee00000-369ee07000 r-xp 00000000 fd:01 5175
369ee07000-369f006000 ---p 00007000 fd:01 5175
369f006000-369f007000 r--p 00006000 fd:01 5175
369f007000-369f008000 rw-p 00007000 fd:01 5175
369f200000-369f283000 r-xp 00000000 fd:01 5182
369f283000-369f482000 ---p 00083000 fd:01 5182
369f482000-369f483000 r--p 00082000 fd:01 5182
369f483000-369f484000 rw-p 00083000 fd:01 5182
369fa00000-369fa0c000 r-xp 00000000 fd:01 5170
369fa0c000-369fc0c000 ---p 0000c000 fd:01 5170
369fc0c000-369fc0d000 r--p 0000c000 fd:01 5170
369fc0d000-369fc0e000 rw-p 0000d000 fd:01 5170
369fe00000-369fe1d000 r-xp 00000000 fd:01 5179
369fe1d000-36a001c000 ---p 0001d000 fd:01 5179
36a001c000-36a001d000 r--p 0001c000 fd:01 5179
36a001d000-36a001e000 rw-p 0001d000 fd:01 5179
36a001e000-36a001f000 rw-p 00000000 00:00 0
36a0200000-36a0219000 r-xp 00000000 fd:08 17924
36a0219000-36a0418000 ---p 00019000 fd:08 17924
36a0418000-36a0419000 r--p 00018000 fd:08 17924
36a0419000-36a041a000 rw-p 00019000 fd:08 17924
36a0600000-36a0616000 r-xp 00000000 fd:01 3772
36a0616000-36a0815000 ---p 00016000 fd:01 3772
36a0815000-36a0816000 rw-p 00015000 fd:01 3772
36a0a00000-36a0a16000 r-xp 00000000 fd:01 5178
36a0a16000-36a0c16000 ---p 00016000 fd:01 5178
36a0c16000-36a0c17000 r--p 00016000 fd:01 5178
36a0c17000-36a0c18000 rw-p 00017000 fd:01 5178
36a0c18000-36a0c1a000 rw-p 00000000 00:00 0
36a0e00000-36a0e49000 r-xp 00000000 fd:01 1926
36a0e49000-36a1048000 ---p 00049000 fd:01 1926
36a1048000-36a1049000 r--p 00048000 fd:01 1926
36a1049000-36a104b000 rw-p 00049000 fd:01 1926
36a1600000-36a160e000 r-xp 00000000 fd:01 3777
36a160e000-36a180d000 ---p 0000e000 fd:01 3777
36a180d000-36a180e000 r--p 0000d000 fd:01 3777
36a180e000-36a180f000 rw-p 0000e000 fd:01 3777
36a1a00000-36a1a5d000 r-xp 00000000 fd:01 5168
36a1a5d000-36a1c5c000 ---p 0005d000 fd:01 5168
36a1c5c000-36a1c5d000 r--p 0005c000 fd:01 5168
36a1c5d000-36a1c5e000 rw-p 0005d000 fd:01 5168
36a1c5e000-36a1c62000 rw-p 00000000 00:00 0
36a1e00000-36a1e07000 r-xp 00000000 fd:01 5169
36a1e07000-36a2007000 ---p 00007000 fd:01 5169
36a2007000-36a2008000 r--p 00007000 fd:01 5169
36a2008000-36a2009000 rw-p 00008000 fd:01 5169
36a2009000-36a2037000 rw-p 00000000 00:00 0
36a2200000-36a2203000 r-xp 00000000 fd:01 5192
36a2203000-36a2402000 ---p 00003000 fd:01 5192
36a2402000-36a2403000 r--p 00002000 fd:01 5192
36a2403000-36a2404000 rw-p 00003000 fd:01 5192
36a2600000-36a2620000 r-xp 00000000 fd:08 5902
36a2620000-36a281f000 ---p 00020000 fd:08 5902
36a281f000-36a2825000 r--p 0001f000 fd:08 5902
36a2825000-36a2826000 rw-p 00025000 fd:08 5902
36a2a00000-36a2b47000 r-xp 00000000 fd:08 18039
36a2b47000-36a2d46000 ---p 00147000 fd:08 18039
36a2d46000-36a2d50000 rw-p 00146000 fd:08 18039
36a2d50000-36a2d51000 rw-p 00000000 00:00 0
36a2e00000-36a2e03000 r-xp 00000000 fd:01 5188
36a2e03000-36a3002000 ---p 00003000 fd:01 5188
36a3002000-36a3003000 r--p 00002000 fd:01 5188
36a3003000-36a3004000 rw-p 00003000 fd:01 5188
36a3200000-36a3239000 r-xp 00000000 fd:01 663
36a3239000-36a3438000 ---p 00039000 fd:01 663
36a3438000-36a3439000 r--p 00038000 fd:01 663
36a3439000-36a343b000 rw-p 00039000 fd:01 663
36a343b000-36a343d000 rw-p 00000000 00:00 0
36a3600000-36a3638000 r-xp 00000000 fd:08 21660
36a3638000-36a3838000 ---p 00038000 fd:08 21660
36a3838000-36a383a000 r--p 00038000 fd:08 21660
36a383a000-36a383b000 rw-p 0003a000 fd:08 21660
36a383b000-36a383c000 rw-p 00000000 00:00 0
36a3a00000-36a3a04000 r-xp 00000000 fd:01 5187
36a3a04000-36a3c03000 ---p 00004000 fd:01 5187
36a3c03000-36a3c04000 r--p 00003000 fd:01 5187
36a3c04000-36a3c05000 rw-p 00004000 fd:01 5187
36a3e00000-36a3f33000 r-xp 00000000 fd:08 21659
36a3f33000-36a4132000 ---p 00133000 fd:08 21659
36a4132000-36a4137000 r--p 00132000 fd:08 21659
36a4137000-36a4139000 rw-p 00137000 fd:08 21659
36a4139000-36a413b000 rw-p 00000000 00:00 0
36a4200000-36a43ba000 r-xp 00000000 fd:08 20219
36a43ba000-36a45b9000 ---p 001ba000 fd:08 20219
36a45b9000-36a45d4000 r--p 001b9000 fd:08 20219
36a45d4000-36a45e0000 rw-p 001d4000 fd:08 20219
36a45e0000-36a45e4000 rw-p 00000000 00:00 0
36a4600000-36a4628000 r-xp 00000000 fd:08 20218
36a4628000-36a4828000 ---p 00028000 fd:08 20218
36a4828000-36a482b000 r--p 00028000 fd:08 20218
36a482b000-36a482c000 rw-p 0002b000 fd:08 20218
36a4a00000-36a4a02000 r-xp 00000000 fd:01 237
36a4a02000-36a4c01000 ---p 00002000 fd:01 237
36a4c01000-36a4c02000 r--p 00001000 fd:01 237
36a4c02000-36a4c03000 rw-p 00002000 fd:01 237
36a4e00000-36a4e0a000 r-xp 00000000 fd:01 5190
36a4e0a000-36a5009000 ---p 0000a000 fd:01 5190
36a5009000-36a500a000 r--p 00009000 fd:01 5190
36a500a000-36a500b000 rw-p 0000a000 fd:01 5190
36a5200000-36a52d4000 r-xp 00000000 fd:01 5193
36a52d4000-36a54d4000 ---p 000d4000 fd:01 5193
36a54d4000-36a54dd000 r--p 000d4000 fd:01 5193
36a54dd000-36a54df000 rw-p 000dd000 fd:01 5193
36a5600000-36a563f000 r-xp 00000000 fd:01 5194
36a563f000-36a583f000 ---p 0003f000 fd:01 5194
36a583f000-36a5840000 r--p 0003f000 fd:01 5194
36a5840000-36a5842000 rw-p 00040000 fd:01 5194
36a5a00000-36a5a51000 r-xp 00000000 fd:08 6885
36a5a51000-36a5c50000 ---p 00051000 fd:08 6885
36a5c50000-36a5c53000 rw-p 00050000 fd:08 6885
36a5e00000-36a5e2a000 r-xp 00000000 fd:01 5191
36a5e2a000-36a6029000 ---p 0002a000 fd:01 5191
36a6029000-36a602b000 r--p 00029000 fd:01 5191
36a602b000-36a602c000 rw-p 0002b000 fd:01 5191
36a6200000-36a6226000 r-xp 00000000 fd:08 16520
36a6226000-36a6426000 ---p 00026000 fd:08 16520
36a6426000-36a6427000 rw-p 00026000 fd:08 16520
36a7200000-36a7262000 r-xp 00000000 fd:08 17699
36a7262000-36a7461000 ---p 00062000 fd:08 17699
36a7461000-36a7465000 r--p 00061000 fd:08 17699
36a7465000-36a746c000 rw-p 00065000 fd:08 17699
7f61c8000000-7f61c8021000 rw-p 00000000
7f61c8021000-7f61cc000000 ---p 00000000
7f61ccdba000-7f61ccdc0000 r-xp 00000000 fd:08 266931
7f61ccdc0000-7f61ccfbf000 ---p 00006000 fd:08 266931
7f61ccfbf000-7f61ccfc0000 rw-p 00005000 fd:08 266931
7f61ccfc0000-7f61cd1df000 rw-p 00000000
7f61cd1df000-7f61cd1eb000 r-xp 00000000 fd:01 3754
7f61cd1eb000-7f61cd3eb000 ---p 0000c000 fd:01 3754
7f61cd3eb000-7f61cd3ec000 r--p 0000c000 fd:01 3754
7f61cd3ec000-7f61cd3ed000 rw-p 0000d000 fd:01 3754
7f61cd3f7000-7f61cd406000 rw-s 00000000 00:10 1491041
7f61cd406000-7f63a0bd6000 rw-s 00000000 00:04 1491038
7f63a0bd6000-7f63a0bd9000 r-xp 00000000 fd:08 268834
7f63a0bd9000-7f63a0dd8000 ---p 00003000 fd:08 268834
7f63a0dd8000-7f63a0dd9000 rw-p 00002000 fd:08 268834
7f63a0dd9000-7f63a0dda000 r-xp 00000000 fd:08 266927
7f63a0dda000-7f63a0fd9000 ---p 00001000 fd:08 266927
7f63a0fd9000-7f63a0fda000 rw-p 00000000 fd:08 266927
7f63a0fda000-7f63a0fdc000 r-xp 00000000 fd:08 266928
7f63a0fdc000-7f63a11dc000 ---p 00002000 fd:08 266928
7f63a11dc000-7f63a11dd000 rw-p 00002000 fd:08 266928
7f63a11dd000-7f63a11e5000 r-xp 00000000 fd:08 266954
7f63a11e5000-7f63a13e4000 ---p 00008000 fd:08 266954
7f63a13e4000-7f63a13e5000 rw-p 00007000 fd:08 266954
7f63a13e5000-7f63a1433000 r-xp 00000000 fd:01 644
7f63a1433000-7f63a1633000 ---p 0004e000 fd:01 644
7f63a1633000-7f63a1634000 r--p 0004e000 fd:01 644
7f63a1634000-7f63a1636000 rw-p 0004f000 fd:01 644
7f63a1636000-7f63a1638000 rw-p 00000000
7f63a1638000-7f63a166a000 r-xp 00000000 fd:01 5189
7f63a166a000-7f63a1869000 ---p 00032000 fd:01 5189
7f63a1869000-7f63a186a000 rw-p 00031000 fd:01 5189
7f63a186a000-7f63a18af000 r-xp 00000000 fd:08 266281
7f63a18af000-7f63a1aaf000 ---p 00045000 fd:08 266281
7f63a1aaf000-7f63a1ab2000 rw-p 00045000 fd:08 266281
7f63a1ab2000-7f63a1b5a000 r-xp 00000000 fd:08 268586
7f63a1b5a000-7f63a1d5a000 ---p 000a8000 fd:08 268586
7f63a1d5a000-7f63a1d5e000 rw-p 000a8000 fd:08 268586
7f63a1d5e000-7f63a1d64000 rw-p 00000000
7f63a1d64000-7f63a7bf5000 r--p 00000000 fd:08 21655
7f63a7bf5000-7f63a7bfd000 rw-p 00000000
7f63a7bfd000-7f63a7ce5000 r-xp 00000000 fd:08 1530
7f63a7ce5000-7f63a7ee5000 ---p 000e8000 fd:08 1530
7f63a7ee5000-7f63a7eec000 r--p 000e8000 fd:08 1530
7f63a7eec000-7f63a7eee000 rw-p 000ef000 fd:08 1530
7f63a7eee000-7f63a7f03000 rw-p 00000000
7f63a7f03000-7f63a8e48000 r-xp 00000000 fd:08 22589
7f63a8e48000-7f63a9047000 ---p 00f45000 fd:08 22589
7f63a9047000-7f63a9048000 rw-p 00f44000 fd:08 22589
7f63a9048000-7f63a904e000 rw-p 00000000
7f63a904e000-7f63a9065000 r-xp 00000000 fd:01 5166
7f63a9065000-7f63a9264000 ---p 00017000 fd:01 5166
7f63a9264000-7f63a9265000 r--p 00016000 fd:01 5166
7f63a9265000-7f63a926a000 rw-p 00017000 fd:01 5166
7f63a926a000-7f63a927f000 r-xp 00000000 fd:01 5181
7f63a927f000-7f63a947e000 ---p 00015000 fd:01 5181
7f63a947e000-7f63a947f000 r--p 00014000 fd:01 5181
7f63a947f000-7f63a9480000 rw-p 00015000 fd:01 5181
7f63a9480000-7f63a9481000 rw-p 00000000
7f63a9481000-7f63a95c0000 r-xp 00000000 fd:08 22601
7f63a95c0000-7f63a97c0000 ---p 0013f000 fd:08 22601
7f63a97c0000-7f63a97d1000 rw-p 0013f000 fd:08 22601
7f63a97d1000-7f63a97d3000 rw-p 00000000
7f63a97d3000-7f63a995b000 r-xp 00000000 fd:08 22591
7f63a995b000-7f63a9b5b000 ---p 00188000 fd:08 22591
7f63a9b5b000-7f63a9b68000 rw-p 00188000 fd:08 22591
7f63a9b68000-7f63a9b6d000 rw-p 00000000
7f63a9b75000-7f63a9b76000 rw-p 00000000
7f63a9b76000-7f63a9b77000 rw-s 00000000 00:04 262145
7f63a9b77000-7f63a9b78000 rw-p 00000000
7fff4a3fc000-7fff4a411000 rw-p 00000000 00:00 0
7fff4a4c3000-7fff4a4c4000 r-xp 00000000 00:00 0
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
2019-11-24 11:07:53.735 CST 14713 127.0.0.1(39348)
2019-11-24 11:07:53.735 CST 14713 127.0.0.1(39348)
2019-11-24 11:07:53.735 CST 14713 127.0.0.1(39348)
2019-11-24 11:07:53.735 CST 14713 127.0.0.1(39348)
2019-11-24 11:07:53.751 CST 5930 127.0.0.1(38471)
2019-11-24 11:07:53.751 CST 5930 127.0.0.1(38471)
2019-11-24 11:07:53.751 CST 5930 127.0.0.1(38471)
2019-11-24 11:07:53.767 CST 14710 127.0.0.1(39345)
2019-11-24 11:07:53.767 CST 14710 127.0.0.1(39345)
2019-11-24 11:07:53.767 CST 14710 127.0.0.1(39345)
2019-11-24 11:07:53.776 CST 13352 127.0.0.1(39181)
2019-11-24 11:07:53.776 CST 13352 127.0.0.1(39181)
2019-11-24 11:07:53.776 CST 13352 127.0.0.1(39181)
2019-11-24 11:08:09.854 CST 14668
2019-11-24 11:08:09.854 CST 14668
2019-11-24 11:08:09.854 CST 14668 127.0.0.1(39349) COMMIT PREPARED: double free or corruption (!prev): 0x0000000001a977a0 *** COMMIT PREPARED(XLogReaderFree+0x57)[0x4ff947] COMMIT PREPARED[0x4e4387] COMMIT PREPARED(FinishPreparedTransaction+0x139)[0x4e5849] COMMIT PREPARED(standard_ProcessUtility+0x711)[0x72d601] y+0x741)[0x7f63a1ae97e1] 8)[0x7f63a11e1178] COMMIT PREPARED[0x729388] COMMIT PREPARED[0x72a2fd] COMMIT PREPARED(PortalRun+0x238)[0x72aa98] COMMIT PREPARED[0x727051] COMMIT PREPARED(PostgresMain+0x549)[0x728039] COMMIT PREPARED(PostmasterMain+0x194a)[0x6bb43a] COMMIT PREPARED(main+0x7c0)[0x63b4d0] e21ed5d] COMMIT PREPARED[0x477149] /usr/pgsql-10/bin/postgres /usr/pgsql-10/bin/postgres /lib64/ld-2.12.so /lib64/ld-2.12.so /lib64/ld-2.12.so /lib64/libc-2.12.so /lib64/libc-2.12.so /lib64/libc-2.12.so /lib64/libc-2.12.so /lib64/libdl-2.12.so /lib64/libdl-2.12.so /lib64/libdl-2.12.so /lib64/libdl-2.12.so /lib64/libpthread-2.12.so /lib64/libpthread-2.12.so /lib64/libpthread-2.12.so /lib64/libpthread-2.12.so /lib64/librt-2.12.so /lib64/librt-2.12.so /lib64/librt-2.12.so /lib64/librt-2.12.so /lib64/libm-2.12.so /lib64/libm-2.12.so /lib64/libm-2.12.so /lib64/libm-2.12.so /lib64/libpam.so.0.82.2 /lib64/libpam.so.0.82.2 /lib64/libpam.so.0.82.2 /lib64/libpam.so.0.82.2 /lib64/libselinux.so.1 /lib64/libselinux.so.1 /lib64/libselinux.so.1 /lib64/libselinux.so.1 /usr/lib64/libsasl2.so.2.0.23 /usr/lib64/libsasl2.so.2.0.23 /usr/lib64/libsasl2.so.2.0.23 /usr/lib64/libsasl2.so.2.0.23 /lib64/libgcc_s-4.4.6-20120305.so.1 /lib64/libgcc_s-4.4.6-20120305.so.1 /lib64/libgcc_s-4.4.6-20120305.so.1 /lib64/libresolv-2.12.so /lib64/libresolv-2.12.so /lib64/libresolv-2.12.so /lib64/libresolv-2.12.so /lib64/libldap-2.4.so.2.5.6 /lib64/libldap-2.4.so.2.5.6 /lib64/libldap-2.4.so.2.5.6 /lib64/libldap-2.4.so.2.5.6 /lib64/liblber-2.4.so.2.5.6 /lib64/liblber-2.4.so.2.5.6 /lib64/liblber-2.4.so.2.5.6 /lib64/liblber-2.4.so.2.5.6 /lib64/libfreebl3.so /lib64/libfreebl3.so /lib64/libfreebl3.so /lib64/libfreebl3.so /lib64/libcrypt-2.12.so /lib64/libcrypt-2.12.so /lib64/libcrypt-2.12.so /lib64/libcrypt-2.12.so /lib64/libcom_err.so.2.1 /lib64/libcom_err.so.2.1 /lib64/libcom_err.so.2.1 /lib64/libcom_err.so.2.1 /usr/lib64/libnssutil3.so /usr/lib64/libnssutil3.so /usr/lib64/libnssutil3.so /usr/lib64/libnssutil3.so /usr/lib64/libxml2.so.2.7.6 /usr/lib64/libxml2.so.2.7.6 /usr/lib64/libxml2.so.2.7.6 /lib64/libplds4.so /lib64/libplds4.so /lib64/libplds4.so /lib64/libplds4.so /lib64/libnspr4.so /lib64/libnspr4.so /lib64/libnspr4.so /lib64/libnspr4.so /usr/lib64/libssl3.so /usr/lib64/libssl3.so /usr/lib64/libssl3.so /usr/lib64/libssl3.so /lib64/libplc4.so /lib64/libplc4.so /lib64/libplc4.so /lib64/libplc4.so /usr/lib64/libnss3.so /usr/lib64/libnss3.so /usr/lib64/libnss3.so /usr/lib64/libnss3.so /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libsmime3.so /usr/lib64/libsmime3.so /usr/lib64/libsmime3.so /usr/lib64/libsmime3.so /lib64/libkeyutils.so.1.3 /lib64/libkeyutils.so.1.3 /lib64/libkeyutils.so.1.3 /lib64/libkeyutils.so.1.3 /lib64/libkrb5support.so.0.1 /lib64/libkrb5support.so.0.1 /lib64/libkrb5support.so.0.1 /lib64/libkrb5support.so.0.1 /lib64/libkrb5.so.3.3 /lib64/libkrb5.so.3.3 /lib64/libkrb5.so.3.3 /lib64/libkrb5.so.3.3 /lib64/libgssapi_krb5.so.2.2 /lib64/libgssapi_krb5.so.2.2 /lib64/libgssapi_krb5.so.2.2 /lib64/libgssapi_krb5.so.2.2 /usr/lib64/libcurl.so.4.1.1 /usr/lib64/libcurl.so.4.1.1 /usr/lib64/libcurl.so.4.1.1 /lib64/libk5crypto.so.3.1 /lib64/libk5crypto.so.3.1 /lib64/libk5crypto.so.3.1 /lib64/libk5crypto.so.3.1 /usr/lib64/libssh2.so.1.0.1 /usr/lib64/libssh2.so.1.0.1 /usr/lib64/libssh2.so.1.0.1 /usr/lib64/libssl.so.1.0.1e /usr/lib64/libssl.so.1.0.1e /usr/lib64/libssl.so.1.0.1e /usr/lib64/libssl.so.1.0.1e 00:00 0 00:00 0 /usr/pgsql-10/lib/btree_gin.so /usr/pgsql-10/lib/btree_gin.so /usr/pgsql-10/lib/btree_gin.so 00:00 0 /lib64/libnss_files-2.12.so /lib64/libnss_files-2.12.so /lib64/libnss_files-2.12.so /lib64/libnss_files-2.12.so /dev/shm/PostgreSQL.1869824892 /dev/zero (deleted) /usr/pgsql-10/lib/timescaledb.so /usr/pgsql-10/lib/timescaledb.so /usr/pgsql-10/lib/timescaledb.so /usr/pgsql-10/lib/auth_delay.so /usr/pgsql-10/lib/auth_delay.so /usr/pgsql-10/lib/auth_delay.so /usr/pgsql-10/lib/auto_explain.so /usr/pgsql-10/lib/auto_explain.so /usr/pgsql-10/lib/auto_explain.so /usr/pgsql-10/lib/pg_stat_statements.so /usr/pgsql-10/lib/pg_stat_statements.so /usr/pgsql-10/lib/pg_stat_statements.so /lib64/libldap_r-2.4.so.2.5.6 /lib64/libldap_r-2.4.so.2.5.6 /lib64/libldap_r-2.4.so.2.5.6 /lib64/libldap_r-2.4.so.2.5.6 00:00 0 /lib64/libidn.so.11.6.1 /lib64/libidn.so.11.6.1 /lib64/libidn.so.11.6.1 /usr/pgsql-10/lib/libpq.so.5.10 /usr/pgsql-10/lib/libpq.so.5.10 /usr/pgsql-10/lib/libpq.so.5.10 /usr/pgsql-10/lib/citus.so /usr/pgsql-10/lib/citus.so /usr/pgsql-10/lib/citus.so 00:00 0 /usr/lib/locale/locale-archive 00:00 0 /usr/lib64/libstdc++.so.6.0.13 /usr/lib64/libstdc++.so.6.0.13 /usr/lib64/libstdc++.so.6.0.13 /usr/lib64/libstdc++.so.6.0.13 00:00 0 /usr/lib64/libicudata.so.42.1 /usr/lib64/libicudata.so.42.1 /usr/lib64/libicudata.so.42.1 00:00 0 /lib64/libaudit.so.1.0.0 /lib64/libaudit.so.1.0.0 /lib64/libaudit.so.1.0.0 /lib64/libaudit.so.1.0.0 /lib64/libz.so.1.2.3 /lib64/libz.so.1.2.3 /lib64/libz.so.1.2.3 /lib64/libz.so.1.2.3 00:00 0 /usr/lib64/libicuuc.so.42.1 /usr/lib64/libicuuc.so.42.1 /usr/lib64/libicuuc.so.42.1 00:00 0 /usr/lib64/libicui18n.so.42.1 /usr/lib64/libicui18n.so.42.1 /usr/lib64/libicui18n.so.42.1 00:00 0 00:00 0 /SYSV0052e2c1 (deleted) 00:00 0 [stack] [vdso] [vsyscall] lobausr lobadbw2 citus - 10.230.239.36:49809 LOG: process 14713 still waiting for ShareLock on transaction 971951123 after 1000.135 ms lobausr lobadbw2 citus - 10.230.239.36:49809 DETAIL: Process holding the lock: 0. Wait queue: 14713. lobausr lobadbw2 citus - 10.230.239.36:49809 CONTEXT: while deleting tuple (88025,10) in relation "loba_tt_ldcs_zcd_undo_103061" lobausr lobadbw2 citus - 10.230.239.36:49809 STATEMENT: DELETE FROM lobauser.loba_tt_ldcs_zcd_undo_103061 loba_tt_ldcs_zcd_undo WHERE (((zcbsf)::text OPERATOR(pg_catalog.=) $1) AND ((exidv)::text OPERATOR(pg_catalog.=) $2)) lobausr lobadbw2 citus - 10.230.239.36:47339 LOG: process 5930 still waiting for AccessExclusiveLock on tuple (88025,10) of relation 129161 of database 16397 after 1000.107 ms lobausr lobadbw2 citus - 10.230.239.36:47339 DETAIL: Process holding the lock: 14713. Wait queue: 5930, 14710, 13352. lobausr lobadbw2 citus - 10.230.239.36:47339 STATEMENT: DELETE FROM lobauser.loba_tt_ldcs_zcd_undo_103061 loba_tt_ldcs_zcd_undo WHERE (((zcbsf)::text OPERATOR(pg_catalog.=) $1) AND ((exidv)::text OPERATOR(pg_catalog.=) $2)) lobausr lobadbw2 citus - 10.230.239.36:49771 LOG: process 14710 still waiting for AccessExclusiveLock on tuple (88025,10) of relation 129161 of database 16397 after 1000.065 ms lobausr lobadbw2 citus - 10.230.239.36:49771 DETAIL: Process holding the lock: 14713. Wait queue: 5930, 14710, 13352. lobausr lobadbw2 citus - 10.230.239.36:49771 STATEMENT: DELETE FROM lobauser.loba_tt_ldcs_zcd_undo_103061 loba_tt_ldcs_zcd_undo WHERE (((zcbsf)::text OPERATOR(pg_catalog.=) $1) AND ((exidv)::text OPERATOR(pg_catalog.=) $2)) lobausr lobadbw2 citus - 10.230.239.36:19373 LOG: process 13352 still waiting for AccessExclusiveLock on tuple (88025,10) of relation 129161 of database 16397 after 1000.071 ms lobausr lobadbw2 citus - 10.230.239.36:19373 DETAIL: Process holding the lock: 14713. Wait queue: 5930, 14710, 13352. lobausr lobadbw2 citus - 10.230.239.36:19373 STATEMENT: DELETE FROM lobauser.loba_tt_ldcs_zcd_undo_103061 loba_tt_ldcs_zcd_undo WHERE (((zcbsf)::text OPERATOR(pg_catalog.=) $1) AND ((exidv)::text OPERATOR(pg_catalog.=) $2)) LOG: server process (PID 14714) was terminated by signal 6: Aborted DETAIL: Failed process was running: COMMIT PREPARED 'citus_0_35791_4207001212_1287199' LOG: terminating any other active server processes

3.2 stacktrace

(gdb) bt
#0 0x000000369e232625 in raise () from /lib64/libc.so.6
#1 0x000000369e233e05 in abort () from /lib64/libc.so.6
#2 0x000000369e270537 in __libc_message () from /lib64/libc.so.6
#3 0x000000369e275f4e in malloc_printerr () from /lib64/libc.so.6
#4 0x000000369e278cf0 in _int_free () from /lib64/libc.so.6
#5 0x00000000004ff947 in XLogReaderFree (state=0x1a403a8) at xlogreader.c:141
#6 0x00000000004e4387 in XlogReadTwoPhaseData (lsn=32886947137584, buf=0x7fff4a40ec38, len=0x0) at twophase.c:1341
#7 0x00000000004e5849 in FinishPreparedTransaction (gid=0x19d7830 "citus_0_35791_4207001212_1287199", isCommit=1 '\001') at twophase.c:1411
#8 0x000000000072d601 in standard_ProcessUtility (pstmt=0x19d7ba0, queryString=0x19d6e48 "COMMIT PREPARED 'citus_0_35791_4207001212_1287199'", context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
queryEnv=0x0, dest=0x19d7c80, completionTag=0x7fff4a40f260 "") at utility.c:460
#9 0x00007f63a1ae97e1 in multi_ProcessUtility (pstmt=0x19d7ba0, queryString=0x19d6e48 "COMMIT PREPARED 'citus_0_35791_4207001212_1287199'", context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
queryEnv=0x0, dest=0x19d7c80, completionTag=0x7fff4a40f260 "") at executor/multi_utility.c:254
#10 0x00007f63a11e1178 in pgss_ProcessUtility (pstmt=0x19d7ba0, queryString=0x19d6e48 "COMMIT PREPARED 'citus_0_35791_4207001212_1287199'", context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
queryEnv=0x0, dest=0x19d7c80, completionTag=0x7fff4a40f260 "") at pg_stat_statements.c:998
#11 0x0000000000729388 in PortalRunUtility (portal=0x1a27128, pstmt=0x19d7ba0, isTopLevel=, setHoldSnapshot=, dest=0x19d7c80,
completionTag=) at pquery.c:1178
#12 0x000000000072a2fd in PortalRunMulti (portal=0x1a27128, isTopLevel=1 '\001', setHoldSnapshot=0 '\000', dest=0x19d7c80, altdest=0x19d7c80, completionTag=0x7fff4a40f260 "")
at pquery.c:1331
#13 0x000000000072aa98 in PortalRun (portal=0x1a27128, count=9223372036854775807, isTopLevel=1 '\001', run_once=1 '\001', dest=0x19d7c80, altdest=0x19d7c80, completionTag=0x7fff4a40f260 "")
at pquery.c:799
#14 0x0000000000727051 in exec_simple_query (query_string=0x19d6e48 "COMMIT PREPARED 'citus_0_35791_4207001212_1287199'") at postgres.c:1122
#15 0x0000000000728039 in PostgresMain (argc=, argv=, dbname=0x1952b48 "lobadbw2", username=) at postgres.c:4117
#16 0x00000000006bb43a in BackendRun (argc=, argv=) at postmaster.c:4405
#17 BackendStartup (argc=, argv=) at postmaster.c:4077
#18 ServerLoop (argc=, argv=) at postmaster.c:1755
#19 PostmasterMain (argc=, argv=) at postmaster.c:1363
#20 0x000000000063b4d0 in main (argc=3, argv=0x190c520) at main.c:228
(gdb) f 5
#5 0x00000000004ff947 in XLogReaderFree (state=0x1a403a8) at xlogreader.c:141
141 xlogreader.c: No such file or directory.
in xlogreader.c
(gdb) p state->readRecordBuf
$3 = 0x1a977d8 "M\001"
(gdb) p state->readRecordBufSize
$7 = 40960
(gdb)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Backend process crashed when executing 2pc transaction in citus

Description

Contributor guide

Assessment

Get new issues in your inbox