citusdata / citusdata/citus

Random failures on `check-follower-cluster ` tests

Open
#2,803 0 comments 0 reactions 0 assignees View on GitHub
regression tests
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

We sometimes get the following error, looks like we've some issues around starting the database in `pg_regress_multi.pl`, but looks a closer look.

```
Makefile:108: recipe for target 'check-follower-cluster' failed
make: *** [check-follower-cluster] Error 25
make: Leaving directory '/home/circleci/project/build-11/src/test/regress'
+ status=2
+ diffs=/home/circleci/project/build-11/src/test/regress/regression.diffs
+ test -f /home/circleci/project/build-11/src/test/regress/regression.diffs
+ cat /home/circleci/project/build-11/src/test/regress/regression.diffs
--- /home/circleci/project/src/test/regress/expected/multi_follower_sanity_check.out 2019-06-26 17:23:23.000000000 +0000
+++ /home/circleci/project/build-11/src/test/regress/results/multi_follower_sanity_check.out 2019-06-26 17:24:33.355877506 +0000
@@ -1,10 +1,3 @@
-- check that the nodes are all in read-only mode and rejecting write queries
\c - - - :follower_master_port
-CREATE TABLE tab (a int);
-ERROR: cannot execute CREATE TABLE in a read-only transaction
-\c - - - :follower_worker_1_port
-CREATE TABLE tab (a int);
-ERROR: cannot execute CREATE TABLE in a read-only transaction
-\c - - - :follower_worker_2_port
-CREATE TABLE tab (a int);
-ERROR: cannot execute CREATE TABLE in a read-only transaction
+\connect: FATAL: database "regression" does not exist

======================================================================

--- /home/circleci/project/src/test/regress/expected/multi_follower_select_statements.out 2019-06-26 17:23:23.000000000 +0000
+++ /home/circleci/project/build-11/src/test/regress/results/multi_follower_select_statements.out 2019-06-26 17:24:33.571880155 +0000
@@ -18,32 +18,31 @@
--------------------------

(1 row)

INSERT INTO the_table (a, b) VALUES (1, 1);
INSERT INTO the_table (a, b) VALUES (1, 2);
-- connect to the follower and check that a simple select query works, the follower
-- is still in the default cluster and will send queries to the primary nodes
\c - - - :follower_master_port
SELECT * FROM the_table;
- a | b
----+---
- 1 | 1
- 1 | 2
-(2 rows)
-
+ERROR: relation "the_table" does not exist
+LINE 1: SELECT * FROM the_table;
+ ^
-- now, connect to the follower but tell it to use secondary nodes. There are no
-- secondary nodes so this should fail.
-- (this is :follower_master_port but substitution doesn't work here)
\c "port=9070 dbname=regression options='-c\ citus.use_secondary_nodes=always'"
SELECT * FROM the_table;
-ERROR: node group 2 does not have a secondary node
+ERROR: relation "the_table" does not exist
+LINE 1: SELECT * FROM the_table;
+ ^
-- add the secondary nodes and try again, the SELECT statement should work this time
\c - - - :master_port
SELECT 1 FROM master_add_node('localhost', :follower_worker_1_port,
groupid => (SELECT groupid FROM pg_dist_node WHERE nodeport = :worker_1_port),
noderole => 'secondary');
?column?
----------
1
(1 row)

@@ -51,49 +50,43 @@
groupid => (SELECT groupid FROM pg_dist_node WHERE nodeport = :worker_2_port),
noderole => 'secondary');
?column?
----------
1
(1 row)

\c "port=9070 dbname=regression options='-c\ citus.use_secondary_nodes=always'"
-- now that we've added secondaries this should work
SELECT * FROM the_table;
- a | b
----+---
- 1 | 1
- 1 | 2
-(2 rows)
-
+ERROR: relation "the_table" does not exist
+LINE 1: SELECT * FROM the_table;
+ ^
SELECT
node_name, node_port
FROM
master_get_active_worker_nodes()
ORDER BY
node_name, node_port;
- node_name | node_port
------------+-----------
- localhost | 9071
- localhost | 9072
-(2 rows)
-
+ERROR: function master_get_active_worker_nodes() does not exist
+LINE 4: master_get_active_worker_nodes()
+ ^
+HINT: No function matches the given name and argument types. You might need to add explicit type casts.
-- okay, now let's play with nodecluster. If we change the cluster of our follower node
-- queries should stat failing again, since there are no worker nodes in the new cluster
\c "port=9070 dbname=regression options='-c\ citus.use_secondary_nodes=always\ -c\ citus.cluster_name=second-cluster'"
-- there are no secondary nodes in this cluster, so this should fail!
SELECT * FROM the_table;
-ERROR: there is a shard placement in node group 2 but there are no nodes in that group
+ERROR: relation "the_table" does not exist
+LINE 1: SELECT * FROM the_table;
+ ^
-- now move the secondary nodes into the new cluster and see that the follower, finally
-- correctly configured, can run select queries involving them
\c - - - :master_port
UPDATE pg_dist_node SET nodecluster = 'second-cluster' WHERE noderole = 'secondary';
\c "port=9070 dbname=regression options='-c\ citus.use_secondary_nodes=always\ -c\ citus.cluster_name=second-cluster'"
SELECT * FROM the_table;
- a | b
----+---
- 1 | 1
- 1 | 2
-(2 rows)
-
+ERROR: relation "the_table" does not exist
+LINE 1: SELECT * FROM the_table;
+ ^
-- clean up after ourselves
\c - - - :master_port
DROP TABLE the_table;

======================================================================

+ exit 2
Exited with code 2
```

Contributor guide

Open the contributing guide

Research direction

Start with pg_regress_multi.pl and the check-follower-cluster target in src/test/regress/Makefile. Reproduce the failure and inspect the multi_follower_sanity_check and multi_follower_select_statements regression outputs. Done means the follower-cluster tests consistently start the regression database and pass with the expected results.

Written by the indexing model from the issue text.

Assessment

Tech stack
perl, postgresql
Domain
databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.