Error trying to remove secondary nodes after a fork
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Ran into this issue with a Cloud customer today. The customer had a cluster with followers, which they then forked. The coordinator was stuck in configuring with the following error:
```
2020-03-06 22:59:35.279 UTC [14395][6/974] : [3-1] [app=psql] ERROR: could not open relation with OID 0
2020-03-06 22:59:35.279 UTC [14395][6/974] : [4-1] [app=psql] STATEMENT:
SELECT
master_remove_node(nodename, nodeport)
FROM
pg_dist_node
WHERE
noderole = 'secondary'
```
Trying to run it in psql, we got the following back-trace:
```
(gdb) bt
#0 errfinish (dummy=dummy@entry=0) at elog.c:411
#1 0x0000000000861dd4 in elog_finish (elevel=elevel@entry=20, fmt=fmt@entry=0x8a61f8 "could not open relation with OID %u") at elog.c:1365
#2 0x000000000049304e in relation_open (relationId=0, lockmode=) at relation.c:62
#3 0x00000000004d6666 in index_open (relationId=, lockmode=) at indexam.c:130
#4 0x00007f4867438522 in DeleteNodeRow (nodePort=5432, nodeName=0x2a444c8 "xxxx")
at metadata/node_metadata.c:1398
#5 RemoveNodeFromCluster (nodePort=5432, nodeName=) at metadata/node_metadata.c:986
#6 master_remove_node (fcinfo=) at metadata/node_metadata.c:250
#7 0x000000000060962b in ExecInterpExpr (state=0x2a01ba8, econtext=0x2a017a0, isnull=) at execExprInterp.c:649
#8 0x0000000000613f5d in ExecEvalExprSwitchContext (isNull=0x7ffd20a86c5f, econtext=0x2a017a0, state=0x2a01ba8)
at ../../../src/include/executor/executor.h:307
#9 ExecProject (projInfo=0x2a01ba0) at ../../../src/include/executor/executor.h:341
#10 ExecScan (node=, accessMtd=0x632af0 , recheckMtd=0x632b70 ) at execScan.c:239
#11 0x000000000060c1e2 in ExecProcNode (node=0x2a01690) at ../../../src/include/executor/executor.h:239
#12 ExecutePlan (execute_once=, dest=0x2a07b70, direction=, numberTuples=0, sendTuples=,
operation=CMD_SELECT, use_parallel_mode=, planstate=0x2a01690, estate=0x2a01450) at execMain.c:1646
#13 standard_ExecutorRun (queryDesc=0x28ff1b0, direction=, count=0, execute_once=) at execMain.c:364
#14 0x00007f486741fc8d in CitusExecutorRun (queryDesc=0x28ff1b0, direction=ForwardScanDirection, count=0, execute_once=true)
at executor/multi_executor.c:177
#15 0x00007f4865c46fee in pgss_ExecutorRun (queryDesc=0x28ff1b0, direction=ForwardScanDirection, count=0, execute_once=)
at pg_stat_statements.c:891
#16 0x000000000074ee1d in PortalRunSelect (portal=portal@entry=0x2978770, forward=forward@entry=true, count=0, count@entry=9223372036854775807,
dest=dest@entry=0x2a07b70) at pquery.c:929
#17 0x0000000000750290 in PortalRun (portal=portal@entry=0x2978770, count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true,
run_once=run_once@entry=true, dest=dest@entry=0x2a07b70, altdest=altdest@entry=0x2a07b70, completionTag=0x7ffd20a87130 "") at pquery.c:770
#18 0x000000000074c1c0 in exec_simple_query (
query_string=0x288a740 "SELECT\n master_remove_node(nodename, nodeport)\nFROM\n pg_dist_node\nWHERE\n noderole = 'secondary';")
at postgres.c:1215
#19 0x000000000074e003 in PostgresMain (argc=, argv=argv@entry=0x290b728, dbname=, username=)
at postgres.c:4236
#20 0x00000000006de048 in BackendRun (port=0x28f1460, port=0x28f1460) at postmaster.c:4437
```
```
citus=# select citus_version();
citus_version
----------------------------------------------------------------------------------------------------------------
Citus Enterprise 9.1.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-23), 64-bit
(1 row)
citus=# select version();
version
--------------------------------------------------------------------------------------------------------
PostgreSQL 12.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 7.2.1 20170915 (Red Hat 7.2.1-2), 64-bit
(1 row)
```
Contributor guide
Assessment
This issue has not been assessed yet.