Random failures on `check-follower-cluster ` tests
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
We sometimes get the following error, looks like we've some issues around starting the database in `pg_regress_multi.pl`, but looks a closer look.
```
Makefile:108: recipe for target 'check-follower-cluster' failed
make: *** [check-follower-cluster] Error 25
make: Leaving directory '/home/circleci/project/build-11/src/test/regress'
+ status=2
+ diffs=/home/circleci/project/build-11/src/test/regress/regression.diffs
+ test -f /home/circleci/project/build-11/src/test/regress/regression.diffs
+ cat /home/circleci/project/build-11/src/test/regress/regression.diffs
--- /home/circleci/project/src/test/regress/expected/multi_follower_sanity_check.out 2019-06-26 17:23:23.000000000 +0000
+++ /home/circleci/project/build-11/src/test/regress/results/multi_follower_sanity_check.out 2019-06-26 17:24:33.355877506 +0000
@@ -1,10 +1,3 @@
-- check that the nodes are all in read-only mode and rejecting write queries
\c - - - :follower_master_port
-CREATE TABLE tab (a int);
-ERROR: cannot execute CREATE TABLE in a read-only transaction
-\c - - - :follower_worker_1_port
-CREATE TABLE tab (a int);
-ERROR: cannot execute CREATE TABLE in a read-only transaction
-\c - - - :follower_worker_2_port
-CREATE TABLE tab (a int);
-ERROR: cannot execute CREATE TABLE in a read-only transaction
+\connect: FATAL: database "regression" does not exist
======================================================================
--- /home/circleci/project/src/test/regress/expected/multi_follower_select_statements.out 2019-06-26 17:23:23.000000000 +0000
+++ /home/circleci/project/build-11/src/test/regress/results/multi_follower_select_statements.out 2019-06-26 17:24:33.571880155 +0000
@@ -18,32 +18,31 @@
--------------------------
(1 row)
INSERT INTO the_table (a, b) VALUES (1, 1);
INSERT INTO the_table (a, b) VALUES (1, 2);
-- connect to the follower and check that a simple select query works, the follower
-- is still in the default cluster and will send queries to the primary nodes
\c - - - :follower_master_port
SELECT * FROM the_table;
- a | b
----+---
- 1 | 1
- 1 | 2
-(2 rows)
-
+ERROR: relation "the_table" does not exist
+LINE 1: SELECT * FROM the_table;
+ ^
-- now, connect to the follower but tell it to use secondary nodes. There are no
-- secondary nodes so this should fail.
-- (this is :follower_master_port but substitution doesn't work here)
\c "port=9070 dbname=regression options='-c\ citus.use_secondary_nodes=always'"
SELECT * FROM the_table;
-ERROR: node group 2 does not have a secondary node
+ERROR: relation "the_table" does not exist
+LINE 1: SELECT * FROM the_table;
+ ^
-- add the secondary nodes and try again, the SELECT statement should work this time
\c - - - :master_port
SELECT 1 FROM master_add_node('localhost', :follower_worker_1_port,
groupid => (SELECT groupid FROM pg_dist_node WHERE nodeport = :worker_1_port),
noderole => 'secondary');
?column?
----------
1
(1 row)
@@ -51,49 +50,43 @@
groupid => (SELECT groupid FROM pg_dist_node WHERE nodeport = :worker_2_port),
noderole => 'secondary');
?column?
----------
1
(1 row)
\c "port=9070 dbname=regression options='-c\ citus.use_secondary_nodes=always'"
-- now that we've added secondaries this should work
SELECT * FROM the_table;
- a | b
----+---
- 1 | 1
- 1 | 2
-(2 rows)
-
+ERROR: relation "the_table" does not exist
+LINE 1: SELECT * FROM the_table;
+ ^
SELECT
node_name, node_port
FROM
master_get_active_worker_nodes()
ORDER BY
node_name, node_port;
- node_name | node_port
------------+-----------
- localhost | 9071
- localhost | 9072
-(2 rows)
-
+ERROR: function master_get_active_worker_nodes() does not exist
+LINE 4: master_get_active_worker_nodes()
+ ^
+HINT: No function matches the given name and argument types. You might need to add explicit type casts.
-- okay, now let's play with nodecluster. If we change the cluster of our follower node
-- queries should stat failing again, since there are no worker nodes in the new cluster
\c "port=9070 dbname=regression options='-c\ citus.use_secondary_nodes=always\ -c\ citus.cluster_name=second-cluster'"
-- there are no secondary nodes in this cluster, so this should fail!
SELECT * FROM the_table;
-ERROR: there is a shard placement in node group 2 but there are no nodes in that group
+ERROR: relation "the_table" does not exist
+LINE 1: SELECT * FROM the_table;
+ ^
-- now move the secondary nodes into the new cluster and see that the follower, finally
-- correctly configured, can run select queries involving them
\c - - - :master_port
UPDATE pg_dist_node SET nodecluster = 'second-cluster' WHERE noderole = 'secondary';
\c "port=9070 dbname=regression options='-c\ citus.use_secondary_nodes=always\ -c\ citus.cluster_name=second-cluster'"
SELECT * FROM the_table;
- a | b
----+---
- 1 | 1
- 1 | 2
-(2 rows)
-
+ERROR: relation "the_table" does not exist
+LINE 1: SELECT * FROM the_table;
+ ^
-- clean up after ourselves
\c - - - :master_port
DROP TABLE the_table;
======================================================================
+ exit 2
Exited with code 2
```
Contributor guide
Research direction
Start with pg_regress_multi.pl and the check-follower-cluster target in src/test/regress/Makefile. Reproduce the failure and inspect the multi_follower_sanity_check and multi_follower_select_statements regression outputs. Done means the follower-cluster tests consistently start the regression database and pass with the expected results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- perl, postgresql
- Domain
- databases, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100