matrixorigin / matrixorigin/matrixone

[Bug]: Restore and clone follow-up TPCC loses multiple backend connections in synchronized bursts

Open
#28,080 1 comment 0 reactions 1 assignee Claimed by @gouhongshen View on GitHub
deferred kind/bug
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] I have checked the existing issues.

### Branch Name

4.2-dev

### Commit ID

`dad3554830240904588d7c0a812f42142c791a9a`

### Other Environment Information

- TKE distributed deployment: 3 CN, 1 TN, 3 LogService, 2 Proxy
- Namespace: `mo-snapshot-commit-dad355483-20260902`
- Snapshot/restore nightly run:
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33683918110
- Test shape: snapshot restore or clone followed by concurrent TPCC and
consistency verification

### Actual Behavior

Restore/clone operations reached the follow-up TPCC phase, but multiple
independent jobs then observed clustered backend connection closures and
invalid MySQL connections. New connections subsequently recovered, and every
available TPCC consistency check passed.

This is a transient availability failure, not evidence of persistent TPCC data
corruption.

#### 1. SYS RESTORE NONSYS TO NEWNONSYS

Job:
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33683918110/job/100448321487

```text
TPCC session: 2026-09-02 23:01:22 - 23:03:33 UTC
2026-09-02 23:02:23 - 23:02:38:
15 TT_NEW_ORDER requests:
The connection[...] has not been valid,
caused by: Communications link failure

Transaction Error = 15
Measured tpmE = 6.86
2026-09-02 23:04:48:
Consistency verification successfully.
```

The failures arrived as one burst across many unrelated connection IDs rather
than as one isolated bad session.

#### 2. NONSYS RESTORE NONSYS

Job:
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33683918110/job/100457337320

```text
First TPCC session: 2026-09-03 00:02:01 - 00:04:10 UTC
2026-09-03 00:03:01 - 00:03:36:
16 TT_NEW_ORDER connection failures

Transaction Error = 16
Measured tpmE = 7.42
2026-09-03 00:05:23:
Consistency verification successfully.

Second TPCC session: 2026-09-03 00:06:30 - 00:08:32 UTC
Transaction Error = 0
Measured tpmE = 0.0
2026-09-03 00:09:39:
Consistency verification successfully.
```

The successful second run proves that the cluster recovered without data
repair or recreation. It also rules out a permanently invalid restored
database as the complete explanation.

#### 3. NONSYS CLONE DB AND TABLE TO NONSYS

Job:
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33683918110/job/100474527833

```text
2026-09-03 00:31:53 - 00:32:16:
10 STOCK_LEVEL requests failed with:
ErrorCode 20502, backend connection closed

2026-09-03 00:32:47 - 00:32:58:
3 TT_NEW_ORDER connections became invalid with:
Communications link failure

Transaction Error = 13
Measured tpmE = 6.39
2026-09-03 00:35:41:
Consistency verification successfully.
```

### Current Classification

The observed availability defect is proven, but its internal root cause is not
yet proven.

What is proven:

- failures occur in synchronized bursts across multiple client connections;
- both `backend connection closed` and later invalid JDBC connections are
visible;
- the cluster accepts new work afterward;
- consistency verification succeeds;
- the problem repeats in more than one restore/clone follow-up TPCC job in the
same run.

What is not yet proven:

- which component first closed or stopped serving the connections;
- whether Proxy closed backend connections, a CN listener/session was
unavailable, or CN-to-CN RPC/lock traffic stalled first;
- whether the trigger is restore/clone catalog pressure, resource pressure,
lock contention, an MORPC generation transition, or another failure;
- whether current main still reproduces the same failure.

This issue must remain separate from the concurrent CREATE/DROP ACCOUNT
regression. That issue has a proven global `mo_view_refresh FOR UPDATE` queue;
the evidence here does not establish that queue.

### Relation to Existing Fixes

#26888 is symptom-related but not currently proven to be the same failure.
It covered a specific lock-client MORPC control-probe timeout and uncertain-lock
deadlock path, fixed by #26903 and #26925.

The affected 4.2-dev commit `dad3554830` does not contain the main merge
commits for #26903 or #26925. Current main contains both fixes. Therefore:

- those fixes are valid candidates for mitigating this 4.2 incident;
- their presence in main is not sufficient proof that this complete
restore/clone connection-loss pattern is fixed;
- validation must rerun the same restore/clone TPCC jobs and correlate the
first server-side connection transition.

### Expected Behavior

After a successful restore or clone:

- TPCC connections should remain usable under the supported concurrency;
- one slow backend or RPC should not close unrelated client connections in a
synchronized burst;
- if a request cannot be served, it should receive a bounded database error
and the client should be able to reconnect immediately;
- consistency checks should remain successful.

### Steps to Reproduce

```text
1. Deploy the 4.2-dev distributed topology used by run 33683918110.
2. Execute each snapshot/restore or clone workflow.
3. Immediately run the workflow's concurrent TPCC phase.
4. Repeat the same TPCC phase once without another restore.
5. Preserve Proxy and all CN logs from 60 seconds before the first 20502 or
connection-invalid error until 60 seconds after the last error.
6. Run the TPCC consistency checker after each attempt.
```

Required evidence for final root cause:

- client connection ID and first failure timestamp;
- Proxy frontend/backend connection IDs and close reason;
- source and target CN UUIDs, MORPC backend generation, and original RPC error;
- CN pod restart/readiness and listener state;
- lockservice long waits/deadlock graph events in the same window;
- CPU, memory, goroutine, file descriptor, network, and FileService latency;
- exact statement/transaction IDs for at least one failed connection.

A valid fix must make the original failing run green or deterministically
change the first server-side transition while preserving consistency.

### Additional information

- Related symptom issue: https://github.com/matrixorigin/matrixone/issues/26888
- MORPC fix: https://github.com/matrixorigin/matrixone/pull/26903
- uncertain-lock deadlock fix:
https://github.com/matrixorigin/matrixone/pull/26925
- Duplicate searches included `The connection has not been valid restore`,
`Communications link failure TPCC restore`, and the run ID
`33683918110`.
- Impact: transient service unavailability and failed TPCC transactions after
restore/clone. No persistent data inconsistency was observed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.