matrixorigin / matrixorigin/matrixone

[Bug]: DROP DATABASE takes over two hours and loses connection after high-concurrency table churn

Open
#27,575 0 comments 0 reactions 1 assignee Claimed by @aptend View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

`DROP DATABASE` took more than two hours and ended with `ERROR 2013 (HY000): Lost connection to MySQL server during query` after a five-minute high-concurrency create/drop-table workload.

The failure was reported by the `Alter add and drop Unique Index 10 Threads Concurrent Test` step, but the unique-index workload never started. The failure occurred in its database cleanup/setup statement.

This report does not yet classify the behavior as a regression. It records the product failure observed in nightly CI and the currently confirmed execution path.

## Environment

- MatrixOne branch: `4.2-dev`
- MatrixOne commit: `92db870eecaccc6ceea6b42f004170d14daaa6ab`
- Test date: `2026-08-25`
- Deployment: distributed TKE cluster, 3 CN replicas, 1 TN replica, 2 Proxy replicas
- Nightly workflow commit: `1061109b067f3ac238808c4a97c4cd313266056d`

## Steps to reproduce

The observed CI sequence was:

1. Recreate database `concurrent_test`.
2. For five minutes, run these transactions concurrently:
- 50 virtual users: `CREATE TABLE IF NOT EXISTS test_new_{index}(a INT, b VARCHAR, c DATETIME)`
- 20 virtual users: `DROP TABLE IF EXISTS test_new_{index}`
- `{index}` is selected randomly from `1..10000`.
3. Execute:

```sql
DROP DATABASE IF EXISTS concurrent_test;
CREATE DATABASE concurrent_test;
```

The preceding workload completed successfully with:

- create-table attempts: `277866`, errors: `0`
- drop-table attempts: `118456`, errors: `0`

These counters are successful statements, including `IF EXISTS` / `IF NOT EXISTS` no-op statements; they do not prove the exact number of live tables at cleanup time. The configured table-name space limits it to at most 10,000 user tables.

## Actual behavior

The cleanup statement started at `2026-08-25T03:03:42Z` and failed at `2026-08-25T05:03:47Z`:

```text
ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query
Process completed with exit code 1.
```

Because the shell used `-e`, neither creation of the 100 unique-index test tables nor the unique-index concurrency workload was reached.

The immediately following test executed the same `DROP DATABASE; CREATE DATABASE` setup. Its first subsequent table-creation command began about 0.41 seconds later, and all remaining concurrent-test steps completed successfully. The Actions log contains no OOM, panic, or restart signature. Cluster-side logs and profiles are no longer available because the ephemeral namespace was cleaned up.

## Expected behavior

`DROP DATABASE` should complete in a bounded and operationally acceptable time after a supported high-concurrency table-DDL workload, without dropping the client connection.

## Stability and controls

- Observed reproducer: `1/1` nightly occurrence on the stated `4.2-dev` commit.
- Latest official `main`: not yet reproduced.
- Fixed-scale 100/1,000/5,000/10,000-table control: not yet executed.
- Good/bad commit comparison: not yet executed, so this issue is not yet classified as a regression.
- Post-failure observation: the next database cleanup/setup returned quickly and the remaining workload passed, so this was not a persistent full-cluster outage.

## Evidence

- Failed job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/32743566547/job/97665746505
- Failed step: `Alter add and drop Unique Index 10 Threads Concurrent Test`, from `03:03:42Z` to `05:03:47Z`

## Code analysis

Confirmed behavior in `pkg/sql/compile/ddl.go` at the tested commit:

1. `DropDatabase` lists every relation with `database.Relations`.
2. It opens every relation and fetches `TableDefs` to classify user and index tables.
3. It then loops over `deleteTables` and synchronously invokes an internal `DROP TABLE IF EXISTS ...` statement for every user table.
4. Only after all per-table statements finish does it call the engine database delete operation.

Source: https://github.com/matrixorigin/matrixone/blob/92db870eecaccc6ceea6b42f004170d14daaa6ab/pkg/sql/compile/ddl.go#L188-L260

This serial per-table path explains why cleanup cost grows with the number of remaining relations and why the failing client stayed inside the initial `DROP DATABASE` statement.

The exact component that terminated the connection at approximately two hours is not proven by the available Actions log. Proxy/session/network timeout behavior must be checked in a controlled reproduction with cluster logs. The fast follow-up cleanup is consistent with the first statement having completed or nearly completed most server-side cleanup before the client connection was closed, but that remains an inference.

## Regression coverage

After the root cause is fixed, add a deterministic scale test that creates a known number of plain tables, records the remaining relation count, times `DROP DATABASE`, checks that the connection survives, and verifies complete catalog cleanup. Because this is a scale threshold, it belongs in a dedicated nightly/big-data path rather than ordinary BVT.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.