matrixorigin / matrixorigin/matrixone
[Bug]: surviving CN retries killed CN pipeline endpoint for 1000s after replacement is Ready
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
After one CN is killed by Chaos Mesh and its replacement becomes Ready, a newly submitted TPCC consistency query can still hang on a surviving CN because the surviving CN's internal `cn-backend` keeps retrying the killed CN's pipeline endpoint (`:6002`).
The affected query did not fail fast or switch to an active CN. It remained blocked until the JDBC `socketTimeout` expired after 1,000 seconds.
## Environment
- Branch: `4.2-dev`
- Commit: [`05f8a2938df56e8911701fe0276a37a0b2832f22`](https://github.com/matrixorigin/matrixone/commit/05f8a2938df56e8911701fe0276a37a0b2832f22)
- Date: 2026-08-24
- Deployment: Kubernetes, 2 CN / 1 TN(DN) / 3 LogService / 1 Proxy
- Fault injection: Chaos Mesh `PodChaos`, kill one CN
- Workload: TPCC with periodic consistency verification through Proxy
- Source workflow: [mo-nightly-regression run 31928891852](https://github.com/matrixorigin/mo-nightly-regression/actions/runs/31928891852)
## Steps to reproduce
1. Start a distributed MatrixOne cluster with two CNs and Proxy.
2. Run TPCC and periodically execute the TPCC consistency queries through Proxy.
3. Kill one CN using `PodChaos`.
4. Wait until the replacement CN reports Ready.
5. Submit another TPCC consistency query through Proxy.
6. Inspect the Proxy and surviving-CN logs.
The query that hung in this run was:
```sql
(SELECT d_w_id, d_id, d_next_o_id - 1
FROM bmsql_district)
EXCEPT
(SELECT o_w_id, o_d_id, MAX(o_id)
FROM bmsql_oorder
GROUP BY o_w_id, o_d_id);
```
## Actual behavior
Timeline (UTC+8):
- `10:30:22`: `task_kill_cn` killed CN-A.
- `10:30:53`: the replacement CN became Ready.
- `10:31:01`: Proxy built a new connection to the surviving CN-B, and CN-B accepted the connection.
- From `10:31:04` to `10:47:41`: CN-B's `cn-backend` repeatedly tried to connect to the killed CN-A at port `6002`.
- `10:47:41`: the client finally failed with `Communications link failure` after the configured 1,000-second socket timeout.
- `10:48:12`: the next consistency check started.
- `10:48:18`: the next check completed successfully in about 6 seconds.
Sanitized Proxy evidence proving that the affected client was routed to the surviving CN, not to the killed CN:
```text
2026/08/24 02:31:01.733439 +0000 build connection
client->proxy: :1698 -> :6001
proxy->server: :44002 -> :6001
conn ID: 469423
session ID: 13011
```
Sanitized surviving-CN evidence:
```text
2026/08/24 02:31:01.773212 +0000 mo accept connection
connectionId 469423 | account tpcc_test:admin
2026/08/24 02:31:04.257563 +0000 cn-backend
init remote connection failed, retry later
remote: :6002
error: dial tcp4 :6002: i/o timeout
... the same dead endpoint was retried repeatedly ...
2026/08/24 02:47:41.102813 +0000 cn-backend
init remote connection failed, retry later
remote: :6002
error: dial tcp4 :6002: i/o timeout
```
The surviving CN emitted 472 log records referencing the killed CN's `:6002` endpoint during this request.
Client evidence:
```text
2026-08-24 10:47:41 ERROR Communications link failure
The last packet successfully received from the server was 1,000,109 milliseconds ago.
The last packet sent successfully to the server was 1,000,154 milliseconds ago.
```
The CN only canceled and rolled back the statement after the client-side socket timeout closed the session:
```text
2026/08/24 02:47:42 +0000
Failed to newMessageSenderOnClient ... err=context canceled
statement status=fail error=context canceled
```
## Expected behavior
After the killed CN is removed and a replacement CN is Ready, a query accepted by a surviving CN should not keep targeting the killed CN for the lifetime of the client socket timeout.
The query should either:
1. use the refreshed active-CN topology and complete successfully, or
2. fail within a bounded recovery timeout so the client can retry.
## Stability and controls
- Observed occurrence: 1 confirmed occurrence on the live `4.2-dev` Chaos environment. The fault was not reinjected manually because this is a continuously running shared Chaos environment.
- Before injection: consistency checks completed every approximately 30 seconds, normally in 1-7 seconds.
- After the timed-out request: the next consistency check completed successfully in approximately 6 seconds.
- Other workloads recovered and continued producing results.
- No panic, OOM, pod-level persistent failure, or durable data inconsistency was found.
- Not yet reproduced on latest `main`; development analysis and a focused reproducer are requested.
## Code analysis
Confirmed runtime path:
1. Proxy routed the newly opened client connection to the surviving CN.
2. The surviving CN accepted and began the query.
3. `cn-backend` in `morpc/backend.go` continued retrying the killed CN's pipeline address until the client canceled the request.
Hypothesis for development investigation: the distributed-query remote CN target or compute-node topology used by this statement was stale after CN replacement, and the MORPC backend retry loop did not terminate or re-resolve the target after the killed CN was evicted.
Please investigate why a query accepted after the replacement CN was Ready still selected the killed CN, and why this retry path was bounded only by the client's 1,000-second socket timeout.
## Test-harness note
There is a separate test-tool defect that masks this product failure: the deployed `ConsistencyCheck` bytecode logs `SQLException` but exits with status 0, so the Chaos controller records `Step completed successfully`. The checked-in source contains `System.exit(1)`, indicating that the deployed class and source are inconsistent. This does not cause the MO query hang and should be handled separately.
## Regression coverage
Add a multi-CN Chaos regression that:
1. kills one CN while TPCC consistency queries are running;
2. waits for replacement readiness;
3. issues a new distributed query through Proxy;
4. asserts that no surviving CN continues dialing the killed CN endpoint;
5. enforces a bounded query timeout; and
6. verifies TPCC consistency after recovery.
## Related
- #11838: historical whole-cluster hang after CN/DN restart; closed and not the same observed runtime path.
- #14385: `use of closed network connection` log noise; does not cover a user query blocked on a killed CN.
- #25208: stale Proxy `cnTunnels` keys; explicitly states that routing is unaffected, while this issue occurs inside the surviving CN's remote pipeline path.
Contributor guide
Assessment
This issue has not been assessed yet.