matrixorigin / matrixorigin/matrixone

[Bug]: CREATE ACCOUNT globally serializes on the View metadata gate and times out concurrent account operations

Open
#28,079 0 comments 0 reactions 1 assignee Claimed by @aptend View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] I have checked the existing issues.

### Branch Name

main

### Commit ID

Incident: `a2d9070974cd42fc11ac530e52f962a696df9089`

Still present in current main: `ffc8d9f0625c76e87ba967984ce19db2d082b0ba`

### Other Environment Information

- TKE distributed deployment: 3 CN, 1 TN, 3 LogService, 2 Proxy
- Namespace: `mo-main-commit-a2d907097-20260901`
- Workload: `mo-load` concurrent account lifecycle tests
- Run: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33523792810
- Failing job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33523792810/job/100133983379

### Actual Behavior

The normal-concurrency `create_drop_account` case completed, but
`create_drop_account_high` caused account lifecycle operations to wait for
roughly 60-100 seconds and then fail at the client connection timeout.

The high-concurrency case started at `2026-09-02 05:32:36 UTC`:

```text
CREATE side:
VUSER : 50
SUCCESS : 260
RT_MAX : 75574 ms
RT_AVG : 64650.75 ms

DROP side:
VUSER : 20
SUCCESS : 764
ERROR : 37
RT_MAX : 100089 ms
RT_AVG : 4947.66 ms
```

The failed requests were `DROP ACCOUNT IF EXISTS ...` statements. They
reported:

```text
Communications link failure
The last packet successfully received from the server was about 100,000
milliseconds ago.
```

This is not evidence of a network disconnect by itself. It is the client-side
surface after the SQL request receives no response for the JDBC timeout.

### Proven Regression Cause

PR #27370, merge commit `dae2725bef5c81dcb9b5d2fab9a48afe37e2c760`,
added this call inside every `CREATE ACCOUNT` transaction:

```go
inheritViewMetadataRevalidation(ctx, bh, serviceID, newTenantID)
```

Code:

- https://github.com/matrixorigin/matrixone/blob/ffc8d9f0625c76e87ba967984ce19db2d082b0ba/pkg/frontend/authenticate.go#L10610-L10612
- https://github.com/matrixorigin/matrixone/blob/ffc8d9f0625c76e87ba967984ce19db2d082b0ba/pkg/frontend/authenticate.go#L10629-L10652

The function unconditionally executes:

```sql
SELECT rel_id
FROM mo_catalog.mo_tables
WHERE account_id = 0
AND reldatabase = 'mo_catalog'
AND relname = 'mo_view_refresh'
FOR UPDATE;
```

Definition:

- https://github.com/matrixorigin/matrixone/blob/ffc8d9f0625c76e87ba967984ce19db2d082b0ba/pkg/catalog/view_metadata.go#L30-L31

All concurrent account creations therefore contend for the same system-account
catalog row. The lock is held in the surrounding account-creation transaction,
which converts independent tenant creation into one global serial queue.
Concurrent DROP work then overlaps the backed-up account lifecycle workload and
eventually reaches its 100-second client timeout.

CN logs in the incident window contain hundreds of long lock waits on the same
`mo_catalog/mo_view_refresh` row key. Before #27370, comparable concurrent
account runs had no such gate-row wait burst.

`ViewMetadataRefreshEnabled=false` does not avoid the lock. The code executes
the `FOR UPDATE` first and only consults that flag while handling a
missing-table/missing-database error.

No commit between the incident revision and current main changes either
`inheritViewMetadataRevalidation` or `ViewMetadataLifecycleGateSQL`.
Therefore the regression remains reachable on current main.

PR #27370 references #26227, but #26227 tracks stale View schema metadata. It
does not track this account-concurrency regression.

### Expected Behavior

Creating unrelated accounts must not serialize on one global catalog row.
Concurrent CREATE/DROP ACCOUNT operations should either make bounded progress
or return a database error; they must not stall until the client reports a
connection failure.

The View metadata lifecycle must retain its correctness fence without placing
the whole account-creation transaction behind one globally exclusive row lock.

### Steps to Reproduce

```text
1. Deploy a distributed MatrixOne cluster containing PR #27370.
2. Run the normal account concurrency case as a control:
./start.sh -h -c cases/ddl/create_drop_account/ -g
3. Run the high-concurrency case:
./start.sh -h -c cases/ddl/create_drop_account_high/ -g
4. Use 50 CREATE workers and 20 DROP workers, as in run 33523792810.
5. Record CREATE/DROP latency and query CN lockservice logs for the
mo_catalog.mo_view_refresh row.
```

Expected discriminator:

- a revision before `dae2725bef5` does not show the global gate-row wait
queue;
- `dae2725bef5` and current main do;
- a valid fix must remove the queue while preserving View metadata lifecycle
correctness and transaction atomicity.

### Additional information

- Regression-introducing PR: https://github.com/matrixorigin/matrixone/pull/27370
- Related feature issue, not a duplicate: https://github.com/matrixorigin/matrixone/issues/26227
- Duplicate searches included `CREATE ACCOUNT timeout`, `DROP ACCOUNT
timeout`, `mo_view_refresh lock`, `ViewMetadataLifecycleGateSQL`, and
`Communications link failure account`.
- Historical #20223, #20839, #21932, and #26315 have different causes and do
not cover this regression.
- Impact: availability and severe latency regression in account lifecycle
operations. No persistent data corruption was observed in this run.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.