citusdata / citusdata/citus

master_create_empty_shard does not retry if a failure happens after connection establishment

Open
#2,145 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Found while writing tests for #2119.

If Citus is unable to connect to one of the workers it will attempt to connect to the next worker and emit a `WARNING` but successfully create the shard (assuming it can reach `citus.shard_replication_factor` workers). However, if the coordinator successfully establishes a connection with a worker, then a failure on that connection aborts the entire operation, it doesn't retry on another worker like it should.

Here's [the test output](https://github.com/citusdata/citus/blob/1d65b1f6400e50ea5acaef5d42313d7d4c19e6fc/src/test/regress/expected/failure_create_empty_shard_rep_factor_1.out):

````
CREATE TABLE append_tt1(id int);
SELECT create_distributed_table('append_tt1', 'id', 'append');
create_distributed_table
--------------------------

(1 row)

SELECT * FROM pg_dist_shard;
logicalrelid | shardid | shardstorage | shardminvalue | shardmaxvalue
--------------+---------+--------------+---------------+---------------
(0 rows)

SELECT * FROM pg_dist_shard_placement;
shardid | shardstate | shardlength | nodename | nodeport | placementid
---------+------------+-------------+----------+----------+-------------
(0 rows)

SET citus.shard_replication_factor TO 1;
-- reject all connections immediately
SELECT citus.mitmproxy('flow.kill()');
mitmproxy
-----------

(1 row)

-- the first one goes to the other worker, so this succeeds
SELECT master_create_empty_shard('append_tt1');
master_create_empty_shard
---------------------------
100100
(1 row)

-- create a shard, try with our worker first
SELECT master_create_empty_shard('append_tt1');
WARNING: could not connect to node "localhost:57640"
master_create_empty_shard
---------------------------
100101
(1 row)

-- this one goes to the other worker
SELECT master_create_empty_shard('append_tt1');
master_create_empty_shard
---------------------------
100102
(1 row)

-- if we fail the connection after it's been established Citus doesn't recover
-- kill the connection when we send master_apply_shard_ddl_command
SELECT citus.mitmproxy('flow.contains(b"CREATE TABLE").kill()');
mitmproxy
-----------

(1 row)

SELECT master_create_empty_shard('append_tt1');
ERROR: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
CONTEXT: while executing command on localhost:57640
SELECT * FROM pg_dist_shard;
logicalrelid | shardid | shardstorage | shardminvalue | shardmaxvalue
--------------+---------+--------------+---------------+---------------
append_tt1 | 100100 | t | |
append_tt1 | 100101 | t | |
append_tt1 | 100102 | t | |
(3 rows)

SELECT * FROM pg_dist_shard_placement;
shardid | shardstate | shardlength | nodename | nodeport | placementid
---------+------------+-------------+-----------+----------+-------------
100100 | 1 | 0 | localhost | 57637 | 20
100101 | 1 | 0 | localhost | 57637 | 21
100102 | 1 | 0 | localhost | 57637 | 22
(3 rows)
````

Contributor guide

Open the contributing guide

Research direction

Start with the master_create_empty_shard entry point and the failure scenario documented in src/test/regress/expected/failure_create_empty_shard_rep_factor_1.out. Trace the handling of a connection failure during master_apply_shard_ddl_command, especially after connection establishment. Done means the operation retries another worker and creates the shard instead of aborting.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, postgresql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.