matrixorigin / matrixorigin/matrixone

[Bug]: surviving CN retries killed CN pipeline endpoint for 1000s after replacement is Ready

Open
#27,523 3 comments 0 reactions 1 assignee Claimed by @Ariznawlll View on GitHub
kind/bug severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

After one CN is killed by Chaos Mesh and its replacement becomes Ready, a newly submitted TPCC consistency query can still hang on a surviving CN because the surviving CN's internal `cn-backend` keeps retrying the killed CN's pipeline endpoint (`:6002`).

The affected query did not fail fast or switch to an active CN. It remained blocked until the JDBC `socketTimeout` expired after 1,000 seconds.

## Environment

- Branch: `4.2-dev`
- Commit: [`05f8a2938df56e8911701fe0276a37a0b2832f22`](https://github.com/matrixorigin/matrixone/commit/05f8a2938df56e8911701fe0276a37a0b2832f22)
- Date: 2026-08-24
- Deployment: Kubernetes, 2 CN / 1 TN(DN) / 3 LogService / 1 Proxy
- Fault injection: Chaos Mesh `PodChaos`, kill one CN
- Workload: TPCC with periodic consistency verification through Proxy
- Source workflow: [mo-nightly-regression run 31928891852](https://github.com/matrixorigin/mo-nightly-regression/actions/runs/31928891852)

## Steps to reproduce

1. Start a distributed MatrixOne cluster with two CNs and Proxy.
2. Run TPCC and periodically execute the TPCC consistency queries through Proxy.
3. Kill one CN using `PodChaos`.
4. Wait until the replacement CN reports Ready.
5. Submit another TPCC consistency query through Proxy.
6. Inspect the Proxy and surviving-CN logs.

The query that hung in this run was:

```sql
(SELECT d_w_id, d_id, d_next_o_id - 1
FROM bmsql_district)
EXCEPT
(SELECT o_w_id, o_d_id, MAX(o_id)
FROM bmsql_oorder
GROUP BY o_w_id, o_d_id);
```

## Actual behavior

Timeline (UTC+8):

- `10:30:22`: `task_kill_cn` killed CN-A.
- `10:30:53`: the replacement CN became Ready.
- `10:31:01`: Proxy built a new connection to the surviving CN-B, and CN-B accepted the connection.
- From `10:31:04` to `10:47:41`: CN-B's `cn-backend` repeatedly tried to connect to the killed CN-A at port `6002`.
- `10:47:41`: the client finally failed with `Communications link failure` after the configured 1,000-second socket timeout.
- `10:48:12`: the next consistency check started.
- `10:48:18`: the next check completed successfully in about 6 seconds.

Sanitized Proxy evidence proving that the affected client was routed to the surviving CN, not to the killed CN:

```text
2026/08/24 02:31:01.733439 +0000 build connection
client->proxy: :1698 -> :6001
proxy->server: :44002 -> :6001
conn ID: 469423
session ID: 13011
```

Sanitized surviving-CN evidence:

```text
2026/08/24 02:31:01.773212 +0000 mo accept connection
connectionId 469423 | account tpcc_test:admin

2026/08/24 02:31:04.257563 +0000 cn-backend
init remote connection failed, retry later
remote: :6002
error: dial tcp4 :6002: i/o timeout

... the same dead endpoint was retried repeatedly ...

2026/08/24 02:47:41.102813 +0000 cn-backend
init remote connection failed, retry later
remote: :6002
error: dial tcp4 :6002: i/o timeout
```

The surviving CN emitted 472 log records referencing the killed CN's `:6002` endpoint during this request.

Client evidence:

```text
2026-08-24 10:47:41 ERROR Communications link failure
The last packet successfully received from the server was 1,000,109 milliseconds ago.
The last packet sent successfully to the server was 1,000,154 milliseconds ago.
```

The CN only canceled and rolled back the statement after the client-side socket timeout closed the session:

```text
2026/08/24 02:47:42 +0000
Failed to newMessageSenderOnClient ... err=context canceled
statement status=fail error=context canceled
```

## Expected behavior

After the killed CN is removed and a replacement CN is Ready, a query accepted by a surviving CN should not keep targeting the killed CN for the lifetime of the client socket timeout.

The query should either:

1. use the refreshed active-CN topology and complete successfully, or
2. fail within a bounded recovery timeout so the client can retry.

## Stability and controls

- Observed occurrence: 1 confirmed occurrence on the live `4.2-dev` Chaos environment. The fault was not reinjected manually because this is a continuously running shared Chaos environment.
- Before injection: consistency checks completed every approximately 30 seconds, normally in 1-7 seconds.
- After the timed-out request: the next consistency check completed successfully in approximately 6 seconds.
- Other workloads recovered and continued producing results.
- No panic, OOM, pod-level persistent failure, or durable data inconsistency was found.
- Not yet reproduced on latest `main`; development analysis and a focused reproducer are requested.

## Code analysis

Confirmed runtime path:

1. Proxy routed the newly opened client connection to the surviving CN.
2. The surviving CN accepted and began the query.
3. `cn-backend` in `morpc/backend.go` continued retrying the killed CN's pipeline address until the client canceled the request.

Hypothesis for development investigation: the distributed-query remote CN target or compute-node topology used by this statement was stale after CN replacement, and the MORPC backend retry loop did not terminate or re-resolve the target after the killed CN was evicted.

Please investigate why a query accepted after the replacement CN was Ready still selected the killed CN, and why this retry path was bounded only by the client's 1,000-second socket timeout.

## Test-harness note

There is a separate test-tool defect that masks this product failure: the deployed `ConsistencyCheck` bytecode logs `SQLException` but exits with status 0, so the Chaos controller records `Step completed successfully`. The checked-in source contains `System.exit(1)`, indicating that the deployed class and source are inconsistent. This does not cause the MO query hang and should be handled separately.

## Regression coverage

Add a multi-CN Chaos regression that:

1. kills one CN while TPCC consistency queries are running;
2. waits for replacement readiness;
3. issues a new distributed query through Proxy;
4. asserts that no surviving CN continues dialing the killed CN endpoint;
5. enforces a bounded query timeout; and
6. verifies TPCC consistency after recovery.

## Related

- #11838: historical whole-cluster hang after CN/DN restart; closed and not the same observed runtime path.
- #14385: `use of closed network connection` log noise; does not cover a user query blocked on a killed CN.
- #25208: stale Proxy `cnTunnels` keys; explicitly states that routing is unaffected, while this issue occurs inside the surviving CN's remote pipeline path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.