matrixorigin / matrixorigin/matrixone
[Bug]: CREATE ACCOUNT globally serializes on the View metadata gate and times out concurrent account operations
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### Is there an existing issue for the same bug?
- [x] I have checked the existing issues.
### Branch Name
main
### Commit ID
Incident: `a2d9070974cd42fc11ac530e52f962a696df9089`
Still present in current main: `ffc8d9f0625c76e87ba967984ce19db2d082b0ba`
### Other Environment Information
- TKE distributed deployment: 3 CN, 1 TN, 3 LogService, 2 Proxy
- Namespace: `mo-main-commit-a2d907097-20260901`
- Workload: `mo-load` concurrent account lifecycle tests
- Run: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33523792810
- Failing job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33523792810/job/100133983379
### Actual Behavior
The normal-concurrency `create_drop_account` case completed, but
`create_drop_account_high` caused account lifecycle operations to wait for
roughly 60-100 seconds and then fail at the client connection timeout.
The high-concurrency case started at `2026-09-02 05:32:36 UTC`:
```text
CREATE side:
VUSER : 50
SUCCESS : 260
RT_MAX : 75574 ms
RT_AVG : 64650.75 ms
DROP side:
VUSER : 20
SUCCESS : 764
ERROR : 37
RT_MAX : 100089 ms
RT_AVG : 4947.66 ms
```
The failed requests were `DROP ACCOUNT IF EXISTS ...` statements. They
reported:
```text
Communications link failure
The last packet successfully received from the server was about 100,000
milliseconds ago.
```
This is not evidence of a network disconnect by itself. It is the client-side
surface after the SQL request receives no response for the JDBC timeout.
### Proven Regression Cause
PR #27370, merge commit `dae2725bef5c81dcb9b5d2fab9a48afe37e2c760`,
added this call inside every `CREATE ACCOUNT` transaction:
```go
inheritViewMetadataRevalidation(ctx, bh, serviceID, newTenantID)
```
Code:
- https://github.com/matrixorigin/matrixone/blob/ffc8d9f0625c76e87ba967984ce19db2d082b0ba/pkg/frontend/authenticate.go#L10610-L10612
- https://github.com/matrixorigin/matrixone/blob/ffc8d9f0625c76e87ba967984ce19db2d082b0ba/pkg/frontend/authenticate.go#L10629-L10652
The function unconditionally executes:
```sql
SELECT rel_id
FROM mo_catalog.mo_tables
WHERE account_id = 0
AND reldatabase = 'mo_catalog'
AND relname = 'mo_view_refresh'
FOR UPDATE;
```
Definition:
- https://github.com/matrixorigin/matrixone/blob/ffc8d9f0625c76e87ba967984ce19db2d082b0ba/pkg/catalog/view_metadata.go#L30-L31
All concurrent account creations therefore contend for the same system-account
catalog row. The lock is held in the surrounding account-creation transaction,
which converts independent tenant creation into one global serial queue.
Concurrent DROP work then overlaps the backed-up account lifecycle workload and
eventually reaches its 100-second client timeout.
CN logs in the incident window contain hundreds of long lock waits on the same
`mo_catalog/mo_view_refresh` row key. Before #27370, comparable concurrent
account runs had no such gate-row wait burst.
`ViewMetadataRefreshEnabled=false` does not avoid the lock. The code executes
the `FOR UPDATE` first and only consults that flag while handling a
missing-table/missing-database error.
No commit between the incident revision and current main changes either
`inheritViewMetadataRevalidation` or `ViewMetadataLifecycleGateSQL`.
Therefore the regression remains reachable on current main.
PR #27370 references #26227, but #26227 tracks stale View schema metadata. It
does not track this account-concurrency regression.
### Expected Behavior
Creating unrelated accounts must not serialize on one global catalog row.
Concurrent CREATE/DROP ACCOUNT operations should either make bounded progress
or return a database error; they must not stall until the client reports a
connection failure.
The View metadata lifecycle must retain its correctness fence without placing
the whole account-creation transaction behind one globally exclusive row lock.
### Steps to Reproduce
```text
1. Deploy a distributed MatrixOne cluster containing PR #27370.
2. Run the normal account concurrency case as a control:
./start.sh -h -c cases/ddl/create_drop_account/ -g
3. Run the high-concurrency case:
./start.sh -h -c cases/ddl/create_drop_account_high/ -g
4. Use 50 CREATE workers and 20 DROP workers, as in run 33523792810.
5. Record CREATE/DROP latency and query CN lockservice logs for the
mo_catalog.mo_view_refresh row.
```
Expected discriminator:
- a revision before `dae2725bef5` does not show the global gate-row wait
queue;
- `dae2725bef5` and current main do;
- a valid fix must remove the queue while preserving View metadata lifecycle
correctness and transaction atomicity.
### Additional information
- Regression-introducing PR: https://github.com/matrixorigin/matrixone/pull/27370
- Related feature issue, not a duplicate: https://github.com/matrixorigin/matrixone/issues/26227
- Duplicate searches included `CREATE ACCOUNT timeout`, `DROP ACCOUNT
timeout`, `mo_view_refresh lock`, `ViewMetadataLifecycleGateSQL`, and
`Communications link failure account`.
- Historical #20223, #20839, #21932, and #26315 have different causes and do
not cover this regression.
- Impact: availability and severe latency regression in account lifecycle
operations. No persistent data corruption was observed in this run.
Contributor guide
Assessment
This issue has not been assessed yet.