matrixorigin / matrixorigin/matrixone
[Bug]: DROP DATABASE takes over two hours and loses connection after high-concurrency table churn
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
`DROP DATABASE` took more than two hours and ended with `ERROR 2013 (HY000): Lost connection to MySQL server during query` after a five-minute high-concurrency create/drop-table workload.
The failure was reported by the `Alter add and drop Unique Index 10 Threads Concurrent Test` step, but the unique-index workload never started. The failure occurred in its database cleanup/setup statement.
This report does not yet classify the behavior as a regression. It records the product failure observed in nightly CI and the currently confirmed execution path.
## Environment
- MatrixOne branch: `4.2-dev`
- MatrixOne commit: `92db870eecaccc6ceea6b42f004170d14daaa6ab`
- Test date: `2026-08-25`
- Deployment: distributed TKE cluster, 3 CN replicas, 1 TN replica, 2 Proxy replicas
- Nightly workflow commit: `1061109b067f3ac238808c4a97c4cd313266056d`
## Steps to reproduce
The observed CI sequence was:
1. Recreate database `concurrent_test`.
2. For five minutes, run these transactions concurrently:
- 50 virtual users: `CREATE TABLE IF NOT EXISTS test_new_{index}(a INT, b VARCHAR, c DATETIME)`
- 20 virtual users: `DROP TABLE IF EXISTS test_new_{index}`
- `{index}` is selected randomly from `1..10000`.
3. Execute:
```sql
DROP DATABASE IF EXISTS concurrent_test;
CREATE DATABASE concurrent_test;
```
The preceding workload completed successfully with:
- create-table attempts: `277866`, errors: `0`
- drop-table attempts: `118456`, errors: `0`
These counters are successful statements, including `IF EXISTS` / `IF NOT EXISTS` no-op statements; they do not prove the exact number of live tables at cleanup time. The configured table-name space limits it to at most 10,000 user tables.
## Actual behavior
The cleanup statement started at `2026-08-25T03:03:42Z` and failed at `2026-08-25T05:03:47Z`:
```text
ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query
Process completed with exit code 1.
```
Because the shell used `-e`, neither creation of the 100 unique-index test tables nor the unique-index concurrency workload was reached.
The immediately following test executed the same `DROP DATABASE; CREATE DATABASE` setup. Its first subsequent table-creation command began about 0.41 seconds later, and all remaining concurrent-test steps completed successfully. The Actions log contains no OOM, panic, or restart signature. Cluster-side logs and profiles are no longer available because the ephemeral namespace was cleaned up.
## Expected behavior
`DROP DATABASE` should complete in a bounded and operationally acceptable time after a supported high-concurrency table-DDL workload, without dropping the client connection.
## Stability and controls
- Observed reproducer: `1/1` nightly occurrence on the stated `4.2-dev` commit.
- Latest official `main`: not yet reproduced.
- Fixed-scale 100/1,000/5,000/10,000-table control: not yet executed.
- Good/bad commit comparison: not yet executed, so this issue is not yet classified as a regression.
- Post-failure observation: the next database cleanup/setup returned quickly and the remaining workload passed, so this was not a persistent full-cluster outage.
## Evidence
- Failed job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/32743566547/job/97665746505
- Failed step: `Alter add and drop Unique Index 10 Threads Concurrent Test`, from `03:03:42Z` to `05:03:47Z`
## Code analysis
Confirmed behavior in `pkg/sql/compile/ddl.go` at the tested commit:
1. `DropDatabase` lists every relation with `database.Relations`.
2. It opens every relation and fetches `TableDefs` to classify user and index tables.
3. It then loops over `deleteTables` and synchronously invokes an internal `DROP TABLE IF EXISTS ...` statement for every user table.
4. Only after all per-table statements finish does it call the engine database delete operation.
Source: https://github.com/matrixorigin/matrixone/blob/92db870eecaccc6ceea6b42f004170d14daaa6ab/pkg/sql/compile/ddl.go#L188-L260
This serial per-table path explains why cleanup cost grows with the number of remaining relations and why the failing client stayed inside the initial `DROP DATABASE` statement.
The exact component that terminated the connection at approximately two hours is not proven by the available Actions log. Proxy/session/network timeout behavior must be checked in a controlled reproduction with cluster logs. The fast follow-up cleanup is consistent with the first statement having completed or nearly completed most server-side cleanup before the client connection was closed, but that remains an inference.
## Regression coverage
After the root cause is fixed, add a deterministic scale test that creates a known number of plain tables, records the remaining relation count, times `DROP DATABASE`, checks that the connection survives, and verifies complete catalog cleanup. Because this is a scale threshold, it belongs in a dedicated nightly/big-data path rather than ordinary BVT.
Contributor guide
Assessment
This issue has not been assessed yet.