master_create_empty_shard does not retry if a failure happens after connection establishment
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Found while writing tests for #2119.
If Citus is unable to connect to one of the workers it will attempt to connect to the next worker and emit a `WARNING` but successfully create the shard (assuming it can reach `citus.shard_replication_factor` workers). However, if the coordinator successfully establishes a connection with a worker, then a failure on that connection aborts the entire operation, it doesn't retry on another worker like it should.
Here's [the test output](https://github.com/citusdata/citus/blob/1d65b1f6400e50ea5acaef5d42313d7d4c19e6fc/src/test/regress/expected/failure_create_empty_shard_rep_factor_1.out):
````
CREATE TABLE append_tt1(id int);
SELECT create_distributed_table('append_tt1', 'id', 'append');
create_distributed_table
--------------------------
(1 row)
SELECT * FROM pg_dist_shard;
logicalrelid | shardid | shardstorage | shardminvalue | shardmaxvalue
--------------+---------+--------------+---------------+---------------
(0 rows)
SELECT * FROM pg_dist_shard_placement;
shardid | shardstate | shardlength | nodename | nodeport | placementid
---------+------------+-------------+----------+----------+-------------
(0 rows)
SET citus.shard_replication_factor TO 1;
-- reject all connections immediately
SELECT citus.mitmproxy('flow.kill()');
mitmproxy
-----------
(1 row)
-- the first one goes to the other worker, so this succeeds
SELECT master_create_empty_shard('append_tt1');
master_create_empty_shard
---------------------------
100100
(1 row)
-- create a shard, try with our worker first
SELECT master_create_empty_shard('append_tt1');
WARNING: could not connect to node "localhost:57640"
master_create_empty_shard
---------------------------
100101
(1 row)
-- this one goes to the other worker
SELECT master_create_empty_shard('append_tt1');
master_create_empty_shard
---------------------------
100102
(1 row)
-- if we fail the connection after it's been established Citus doesn't recover
-- kill the connection when we send master_apply_shard_ddl_command
SELECT citus.mitmproxy('flow.contains(b"CREATE TABLE").kill()');
mitmproxy
-----------
(1 row)
SELECT master_create_empty_shard('append_tt1');
ERROR: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
CONTEXT: while executing command on localhost:57640
SELECT * FROM pg_dist_shard;
logicalrelid | shardid | shardstorage | shardminvalue | shardmaxvalue
--------------+---------+--------------+---------------+---------------
append_tt1 | 100100 | t | |
append_tt1 | 100101 | t | |
append_tt1 | 100102 | t | |
(3 rows)
SELECT * FROM pg_dist_shard_placement;
shardid | shardstate | shardlength | nodename | nodeport | placementid
---------+------------+-------------+-----------+----------+-------------
100100 | 1 | 0 | localhost | 57637 | 20
100101 | 1 | 0 | localhost | 57637 | 21
100102 | 1 | 0 | localhost | 57637 | 22
(3 rows)
````
Contributor guide
Research direction
Start with the master_create_empty_shard entry point and the failure scenario documented in src/test/regress/expected/failure_create_empty_shard_rep_factor_1.out. Trace the handling of a connection failure during master_apply_shard_ddl_command, especially after connection establishment. Done means the operation retries another worker and creates the shard instead of aborting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, postgresql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100