citusdata / citusdata/citus

shard_replication_factor couldn't work when 1 worker stop

Open
#5,778 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Hello,
**I try to use shard_replication_factor for data replication.**
**I use PostgreSQL 13.6, Citus 10.2-4**
**I have 1 coordinator and 3 workers:**
nodeid | groupid | nodename | nodeport | noderack | hasmetadata | isactive | noderole | nodecluster | metadatasynced | shouldhaveshards
--------+---------+-----------------+----------+----------+-------------+----------+----------+-------------+----------------+------------------
1 | 1 | 137.184.221.190 | 5432 | default | f | t | primary | default | f | t
2 | 2 | 137.184.221.106 | 5432 | default | f | t | primary | default | f | t
3 | 3 | 137.184.221.235 | 5432 | default | f | t | primary | default | f | t

**I set parameter:**
show citus.shard_replication_factor;
citus.shard_replication_factor
__
2
(1 row)

**I create table and distribute it:**
CREATE TABLE t1 (c1 int);
SELECT create_distributed_table('t1', 'c1');
INSERT INTO t1 SELECT generate_series(1,100);

**I stop worker1 and run this commands in coordinator:**
citus_test=# select count(*) from t1;
WARNING: could not issue cancel request
DETAIL: Client error: PQcancel() -- connect() failed: error 111
ERROR: terminating connection due to administrator command
CONTEXT: while executing command on 137.184.221.190:5432

citus_test=# select count(*) from t1;
WARNING: connection to the remote node 137.184.221.190:5432 failed with the following error: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
could not send SSL negotiation packet: Success
count
__
100
(1 row)
citus_test=# insert into t1 select generate_series (1,100);
WARNING: terminating connection due to administrator command
ERROR: failure on connection marked as essential: 137.184.221.190:5432

citus_test=# insert into t1 select generate_series (1,100);
WARNING: connection to the remote node 137.184.221.190:5432 failed with the following error: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
could not send SSL negotiation packet: Success
WARNING: connection to the remote node 137.184.221.190:5432 failed with the following error: server closed the connection unexpectedly
INSERT 0 100

**I start worker1 and stop worker2, and run this commands in coordinator:**
citus_test=# select count(*) from t1;
WARNING: could not issue cancel request
DETAIL: Client error: PQcancel() -- connect() failed: error 111

ERROR: terminating connection due to administrator command
CONTEXT: while executing command on 137.184.221.106:5432
citus_test=# select count(*) from t1;
WARNING: connection to the remote node 137.184.221.106:5432 failed with the following error: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
could not send SSL negotiation packet: Success
count
__
100
(1 row)

citus_test=# insert into t1 select generate_series (1,100);
WARNING: connection to the remote node 137.184.221.106:5432 failed with the following error: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
could not send SSL negotiation packet: Success
ERROR: could not connect to any active placements
citus_test=# insert into t1 select generate_series (1,100);
WARNING: connection to the remote node 137.184.221.106:5432 failed with the following error: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
could not send SSL negotiation packet: Success
ERROR: could not connect to any active placements

**I start worker2 and stop worker3, and run this commands in coordinator:**
citus_test=# select count(*) from t1;
WARNING: could not issue cancel request
DETAIL: Client error: PQcancel() -- connect() failed: error 111

ERROR: terminating connection due to administrator command
CONTEXT: while executing command on 137.184.221.235:5432
citus_test=# select count(*) from t1;
WARNING: connection to the remote node 137.184.221.235:5432 failed with the following error: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
could not send SSL negotiation packet: Success
count
__
100
(1 row)
citus_test=# insert into t1 select generate_series (1,100);
WARNING: connection to the remote node 137.184.221.235:5432 failed with the following error: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
could not send SSL negotiation packet: Success
ERROR: could not connect to any active placements
citus_test=# insert into t1 select generate_series (1,100);
WARNING: connection to the remote node 137.184.221.235:5432 failed with the following error: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
could not send SSL negotiation packet: Success
ERROR: could not connect to any active placements

**As a result:**
When worker1 gone, SELECT and INSERT statements work successfully.
When worker2 OR worker3 gone, SELECT statements can work, but INSERT statements couldn't work. When I set up everything again, sometimes INSERT statements couldn't work another spacific one worker.
Also when a worker gone, "SELECT * FROM pg_dist_node;" and "SELECT * FROM master_get_active_worker_nodes();" statements indicate this worker aktive.
**How can I see it when a worker gone?
And how can I shard_replication_factor when a worker gone?**

Thank you so much.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.