Infinite loop occurs in ConnectionAccessedDifferentPlacement()

One connection in Citus CN hangs when executing the following delete statement. The CPU of the connection backend process has been 100% and cannot be terminated by `pg_terminate_backend ()`, and then we had to restarted postgres by `pg_ctl stop -mi`.

```
DELETE FROM LOBA_BIZ_FOREIGN_INCOME_DATA_UNDO WHERE ZREFORDER1 = :zreforder1; --ZREFORDER1 is not partition field
```
Through pstack and gcore analysis, it is found that an infinite loop appears on the following line of code.

```
static bool
ConnectionAccessedDifferentPlacement(MultiConnection *connection,
ShardPlacement *placement)
{
dlist_iter placementIter;

dlist_foreach(placementIter, &connection->referencedPlacements) // Infinite loop here
...
}
```
Our system has been running for a long time, and the related processing has not been changed. This is the first time such a problem has occurred. Therefore, this is a low probability failure.

The details are as follows:

## Environment:

1. PostgreSQL 10.7
2. Citus 7.4.1
3. CentOS 7.3

## Executed SQL:

```
BEGIN;
SELECT ... FROM FROM LOBA_BIZ_FOREIGN_INCOME_DATA WHERE ZREFORDER1 = :zreforder1; //ZREFORDER1 is not partition field
SELECT ... FROM LOBA_YSSR_DATA_PROCESS_CLEAR WHERE VBELN = :vbeln AND DEAL_STATUS = '1'; //Run N times with different parameters,VBELN is partition field
INSERT INTO LOBA_YSSR_DATA_PROCESS_UNDO (...) VALUES (...); //Run N*M times
DELETE FROM mapping_vbeln_zreforder1_foreign_income WHERE ZREFORDER1 = :zreforder1; //ZREFORDER1 is partition field
DELETE FROM LOBA_YSSR_DATA_PROCESS_CLEAR WHERE dimension = :zreforder1; //dimension is not partition field
DELETE FROM LOBA_BIZ_FOREIGN_INCOME_DATA_UNDO WHERE ZREFORDER1 = :zreforder1; //hung here,zreforder1 is not partition field
COMMIT;
```

information from gcore:
```
#0 0x00007f3bbb52b830 in ConnectionAccessedDifferentPlacement (placement=0x39aa188, connection=0x23c1078) at connection/placement_connection.c:788
788 dlist_foreach(placementIter, &connection->referencedPlacements)
Missing separate debuginfos, use: debuginfo-install audit-libs-2.6.5-3.el7_3.1.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7_3.5.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.centos.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcrypt-1.5.3-13.el7_3.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50.1.2-15.el7.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-11.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 nspr-4.13.1-1.0.el7_3.x86_64 nss-3.28.4-1.2.el7_3.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.28.4-1.0.el7_3.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.2k-16.el7.x86_64 pam-1.1.8-18.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 systemd-libs-219-30.el7_3.9.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x00007f3bbb52b830 in ConnectionAccessedDifferentPlacement (placement=0x39aa188, connection=0x23c1078) at connection/placement_connection.c:788
#1 StartPlacementListConnection (flags=flags@entry=24, placementAccessList=0x39aa708, userName=0x27bdaf0 "lobausr", userName@entry=0x0)
at connection/placement_connection.c:316
#2 0x00007f3bbb55ef32 in OpenTransactionsForAllTasks (taskList=taskList@entry=0x39a1738, connectionFlags=24)
at transaction/multi_shard_transaction.c:121
#3 0x00007f3bbb530b0d in ExecuteModifyTasks (taskList=taskList@entry=0x39a1738, expectResults=expectResults@entry=0 '\000',
paramListInfo=0x2708d88, scanState=scanState@entry=0x281ca38) at executor/multi_router_executor.c:1113
#4 0x00007f3bbb5310f8 in ExecuteMultipleTasks (isModificationQuery=1 '\001', expectResults=0 '\000', taskList=0x39a1738, scanState=0x281ca38)
at executor/multi_router_executor.c:1000
#5 RouterMultiModifyExecScan (node=0x281ca38) at executor/multi_router_executor.c:528
#6 0x00000000005f12c2 in ExecProcNode (node=0x281ca38) at ../../../src/include/executor/executor.h:250
#7 ExecutePlan (execute_once=, dest=0xca42a0 , direction=, numberTuples=0, sendTuples=0 '\000',
operation=CMD_DELETE, use_parallel_mode=, planstate=0x281ca38, estate=0x281c7d8) at execMain.c:1723
#8 standard_ExecutorRun (queryDesc=0x2708df8, direction=, count=0, execute_once=) at execMain.c:364
#9 0x00007f3bba7910fd in pgss_ExecutorRun (queryDesc=0x2708df8, direction=ForwardScanDirection, count=0, execute_once=)
at pg_stat_statements.c:889
#10 0x00007f3bba58b561 in explain_ExecutorRun (queryDesc=0x2708df8, direction=ForwardScanDirection, count=0, execute_once=)
at auto_explain.c:267
#11 0x000000000071dc1a in ProcessQuery (plan=,
sourceText=0x2708cf8 "DELETE \t\t\tFROM \t\t\t LOBA_BIZ_FOREIGN_INCOME_DATA_UNDO \t\t\tWHERE \t\t\t ZREFORDER1 = $1", params=0x2708d88,
queryEnv=0x0, dest=0xca42a0 , completionTag=0x7ffe1fccbc00 "") at pquery.c:161
#12 0x000000000071de57 in PortalRunMulti (portal=portal@entry=0x1f31a68, isTopLevel=isTopLevel@entry=1 '\001',
setHoldSnapshot=setHoldSnapshot@entry=0 '\000', dest=0xca42a0 , dest@entry=0x1fbe3a8, altdest=0xca42a0 ,
altdest@entry=0x1fbe3a8, completionTag=completionTag@entry=0x7ffe1fccbc00 "") at pquery.c:1286
#13 0x000000000071e995 in PortalRun (portal=, count=1, isTopLevel=, run_once=, dest=0x1fbe3a8,
altdest=0x1fbe3a8, completionTag=0x7ffe1fccbc00 "") at pquery.c:799
#14 0x000000000071c3e4 in PostgresMain (argc=, argv=, dbname=, username=)
---Type to continue, or q to quit---
at postgres.c:2007
#15 0x000000000047b246 in BackendRun (port=0x1f3a6b0) at postmaster.c:4405
#16 BackendStartup (port=0x1f3a6b0) at postmaster.c:4077
#17 ServerLoop () at postmaster.c:1755
#18 0x00000000006b25ef in PostmasterMain (argc=argc@entry=1, argv=argv@entry=0x1ee94c0) at postmaster.c:1363
#19 0x000000000047c06f in main (argc=1, argv=0x1ee94c0) at main.c:228

(gdb) p *placement
$1 = {type = {extensible = {type = T_ExtensibleNode, extnodename = 0x7f3bbb59159a "ShardPlacement"}, citus_tag = T_ShardPlacement},
placementId = 12051, shardId = 114058, shardLength = 0, shardState = FILE_FINALIZED, groupId = 1, nodeName = 0x39aa218 "xxx.xxx.xxx.xxx",
nodePort = 6432, partitionMethod = 104 'h', colocationGroupId = 125, representativeValue = 2348810240}
(gdb) p &(&connection->referencedPlacements)->head
$15 = (dlist_node *) 0x23c1290
(gdb) p (&connection->referencedPlacements)->head
$16 = {prev = 0x3657ec0, next = 0x361c340}
(gdb) p *(ConnectionReference *)(0x361c340-40)
$32 = {userName = 0x361c368 "lobausr", connection = 0x23c1078, hadDML = 0 '\000', hadDDL = 0 '\000', colocationGroupId = 94,
representativeValue = 2348810240, placementId = 8083, connectionNode = {prev = 0x23c1290, next = 0x3657d88}}
(gdb) p *(ConnectionReference *)(0x3657d88-40)
$33 = {userName = 0x3657db0 "lobausr", connection = 0x23c1078, hadDML = 1 '\001', hadDDL = 0 '\000', colocationGroupId = 7,
representativeValue = 1040187392, placementId = 864, connectionNode = {prev = 0x361c340, next = 0x3657df0}}
(gdb) p *(ConnectionReference *)(0x3657df0-40)
$34 = {userName = 0x3657e18 "lobausr", connection = 0x23c1078, hadDML = 1 '\001', hadDDL = 0 '\000', colocationGroupId = 6,
representativeValue = 1040187392, placementId = 736, connectionNode = {prev = 0x3657d88, next = 0x3657e58}}
(gdb) p *(ConnectionReference *)(0x3657e58-40)
$35 = {userName = 0x3a54518 "lobausr", connection = 0x24d8c48, hadDML = 1 '\001', hadDDL = 0 '\000', colocationGroupId = 7,
representativeValue = 2113929216, placementId = 896, connectionNode = {prev = 0x361d040, next = 0x24d8e60}}
(gdb) p *(ConnectionReference *)(0x24d8e60-40)
$36 = {userName = 0x0, connection = 0x0, hadDML = 0 '\000', hadDDL = 0 '\000', colocationGroupId = 0, representativeValue = 37489248,
placementId = 33694400, connectionNode = {prev = 0x3657e58, next = 0x361d040}}
(gdb) p *(ConnectionReference *)(0x361d040-40)
$37 = {userName = 0x361d068 "lobausr", connection = 0x24d8c48, hadDML = 0 '\000', hadDDL = 0 '\000', colocationGroupId = 94,
representativeValue = 3422552064, placementId = 8115, connectionNode = {prev = 0x24d8e60, next = 0x3657e58}}
(gdb) p *(ConnectionReference *)(0x3657ec0-40)
$38 = {userName = 0x3657ee8 "lobausr", connection = 0x23c1078, hadDML = 1 '\001', hadDDL = 0 '\000', colocationGroupId = 6,
representativeValue = 2113929216, placementId = 768, connectionNode = {prev = 0x3657e58, next = 0x23c1290}}
```

According to the above information, the linked list pointers of connection->referencedPlacements is incorrect. It is not a valid doubly linked list, and the prev and next pointers of 0x3657e58 may be wrong.

```
0x3657e58<-0x3657ec0<->head(0x23c1290)<->0x361c340<->0x3657d88<->0x3657df0->0x3657e58*(prev:0x361d040)<->0x24d8e60(Corrupted data)<->0x361d040->0x3657e58
```

The shard placements are as following:

```
=#select logicalrelid,p.* from pg_dist_shard_placement p,pg_dist_shard s where p.shardid = s.shardid and placementid in(12051,8083,864,736,896,33694400,8115,768);

logicalrelid | shardid | shardstate | shardlength | nodename | nodeport | placementid
--------------------------------------------+---------+------------+-------------+---------------+----------+-------------
lobauser.loba_yssr_data_process_undo | 102743 | 1 | 0 | xxx.xxx.xxx.157 | 6432 | 736
lobauser.loba_yssr_data_process_undo | 102775 | 1 | 0 | xxx.xxx.xxx.157 | 6432 | 768
lobauser.loba_yssr_data_process_clear | 102871 | 1 | 0 | xxx.xxx.xxx.157 | 6432 | 864
lobauser.loba_yssr_data_process_clear | 102903 | 1 | 0 | xxx.xxx.xxx.157 | 6432 | 896
lobauser.loba_biz_foreign_income_data | 110090 | 1 | 0 | xxx.xxx.xxx.157 | 6432 | 8083
lobauser.loba_biz_foreign_income_data | 110122 | 1 | 0 | xxx.xxx.xxx.157 | 6432 | 8115
lobauser.loba_biz_foreign_income_data_undo | 114058 | 1 | 0 | xxx.xxx.xxx.157 | 6432 | 12051
(7 rows)
```