Unable to stop citus node if coordinator is unavailable. Unpredictable node stop time up to 40 min.
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
If you create a cluster of a coordinator and one worker and place data on the coordinator, then in the event of a worker crash, the coordinator's shutdown process becomes very-very long (decades minutes).
This can be reproduced by trying to perform a soft stop - systemctl stop postgresql
Looks like it's wrong signal handling, MD can`t stop keepalive connection and checkpointer wait to MD finish
There is minimal example:
postgres 16.6 , citus 12.1.6, 13.0.1
1) create cluster
SELECT citus_set_coordinator_host('ip_coord',5432);
SELECT citus_set_node_property('ip_coord',5432,'shouldhaveshards',true);
SELECT citus_add_node('ip_worker',5432);
CREATE TABLE test (id int);
INSERT INTO test SELECT generate_series(1,10000);
SELECT create_distributed_table('test','id',shard_count=>2);
2) stop server ip_coord or close port 5432 on the server
processes before kill:
65803 ? Ss 0:00 /usr/local/pgsql/bin/postgres -D data
65804 ? Ss 0:00 postgres: checkpointer
65805 ? Ss 0:00 postgres: background writer
65807 ? Ss 0:00 postgres: walwriter
65808 ? Ss 0:00 postgres: autovacuum launcher
65809 ? Ss 0:00 postgres: logical replication launcher
65849 ? Ss 0:00 postgres: postgres test [local] idle
65855 ? Ss 0:00 postgres: Citus Maintenance Daemon: 53581/10
65856 ? Ss 0:00 postgres: postgres test 111.111.11.111(50626) idle
65862 ? Ss 0:00 postgres: postgres test 111.111.11.111(54934) idle
65933 ? Ss 0:00 postgres: postgres test 111.111.11.111(33642) idle
66095 pts/1 S+ 0:00 grep --color=auto postgres
3) kill -s SIGINT 65803 65803 #postmaster pid
processes after kill:
65803 ? Ss 0:00 /usr/local/pgsql/bin/postgres -D data0
65804 ? Ss 0:00 postgres: checkpointer
65855 ? Ss 0:00 postgres: Citus Maintenance Daemon: 53581/10
opened connections:
cp 0 260 ubuntu:48208 192.168.72.141:postgresql ESTABLISHED
tcp 164 0 ubuntu:41230 192.168.72.139:postgresql CLOSE_WAI
Contributor guide
Assessment
This issue has not been assessed yet.