matrixorigin / matrixorigin/matrixone
[Bug]: CREATE DATABASE commits but hangs before responding on one CN in a multi-CN cluster
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
在 3 CN + Proxy 的正常 TKE 拓扑中,`CREATE DATABASE` 的 DDL 已在 TN 成功提交后,CN 在返回 MySQL OK 包之前执行内部 metadata 写入时卡住。客户端约 120 秒后收到:
```text
ERROR 2013 (HY000): Lost connection to MySQL server during query (timed out)
```
这不是 DDL 本身执行失败:目标数据库已经存在。故障会使客户端认为创建失败、连接不可用,并让后续大量无关的测试 case 在各自第一条 `CREATE DATABASE` 上超时。
## Environment
- Branch: `main`(CI 在运行开始时解析的官方 main)
- MatrixOne commit: `370c310a994de258ee01e87c31062590a7022f34`
- Date: `2026-09-17`
- Deployment: TKE, 3 CN / 1 TN / Proxy, CN `14C / 55GiB`
- Namespace: `mo-search-commit-370c310a99-20260917`
- Affected coordinator CN: `nightly-regression-dis-tp-cn-7wqk8`
- Test: [mo-nightly-regression run 35208053354 / MOTR job 105160533255](https://github.com/matrixorigin/mo-nightly-regression/actions/runs/35208053354/job/105160533255)
- MOTR commit: `4a1b1b40667a1fdbcef33eb2ed1c6916dfd2b8bf`
The latest `main` has advanced after this CI run; this issue deliberately records the exact official-main commit that failed in CI, rather than claiming an unexecuted later commit was verified.
## Steps to reproduce
1. Deploy a normal multi-CN cluster with Proxy (the observed topology has 3 CN and 1 TN). Do not inject node, network, memory, or storage failures.
2. Connect through Proxy using the MySQL protocol.
3. Repeatedly create a fresh database. Use a different name for each attempt:
```sql
CREATE DATABASE repro_create_db_001;
CREATE DATABASE repro_create_db_002;
CREATE DATABASE repro_create_db_003;
-- continue with new names
```
4. In the affected run, connections routed to one CN consistently hung. A representative statement was:
```sql
CREATE DATABASE motr_issue_27259;
```
5. After the client receives `ERROR 2013`, reconnect and inspect the database/catalog state:
```sql
SHOW DATABASES LIKE 'motr_issue_27259';
SELECT *
FROM mo_catalog.mo_mysql_compatibility_mode
WHERE dat_name = 'motr_issue_27259';
```
## Actual behavior
The following server/client timeline is confirmed for `CREATE DATABASE motr_issue_27259` (UTC):
1. `10:14:46.501`: connection was accepted by CN `nightly-regression-dis-tp-cn-7wqk8`.
2. `10:14:46.684`: TN received `tn.handle.create.database`.
3. `10:14:46.687`: TN committed the DDL; the TN-side DDL latency was about `3.185ms`.
4. The client did not receive an OK response.
5. `10:16:46`: the client-side 120-second query timeout expired and returned `ERROR 2013`.
There were 21 distinct cases with the same `Lost connection` failure in this one run. The log contains 42 text matches because each failure is printed once in the case summary and once as `FIRST FAILURE`.
In the exclusive phase the failure pattern was highly regular: one roughly every three new connections failed after ~120 seconds, while the two intervening cases completed in roughly 0.3–0.7 seconds. For example:
```text
g-04 -> timeout after ~120s
g-05 -> success in ~0.6s
g-06 -> success in ~0.5s
g-07 -> timeout after ~120s
```
The same failure hit unrelated suites (`01_transaction`, `02_gc_checkpoint_flush`, `04_snapshot_pitr`, and `14_issue_regression`) at their initial `CREATE DATABASE`; therefore these are not independent case or baseline failures.
On cancellation, the exclusive queue had completed only `62/197` cases, with `135` not started. Nineteen 120-second connection waits alone consumed about 38 minutes.
## Expected behavior
- Once `CREATE DATABASE` commits successfully, the client must receive an OK response in bounded time.
- Any post-DDL internal metadata failure must return a clear server error; it must not leave the client waiting until a MySQL connection timeout.
- A client-visible timeout must not occur after the DDL has already committed without an unambiguous outcome.
- One unhealthy remote pipeline must not turn independent new connections routed through Proxy into repeated 120-second hangs.
## Stability and controls
- Reproduction evidence: 21 matching client timeouts in one fresh 3-CN CI environment; the timeout/passing cadence correlates with Proxy routing among 3 CN replicas.
- Control: the two CN routing alternatives successfully completed the same initial `CREATE DATABASE` setup in sub-second time; TN committed both successful and failing-route DDL requests in milliseconds.
- Data correctness check: the failing statement had already committed its DDL at TN before the client timeout. The failure is response/lifecycle correctness, not a rejected DDL.
- Resource checks around the failures: no Pod restart increase, no OOMKilled, no panic/fatal evidence, low CPU usage (~0.2–0.43 cores of 14) and low CN memory (~2.1–2.6GiB of 55GiB). This excludes ordinary resource exhaustion as the trigger.
## Evidence from server profiles and logs
The affected coordinator CN was blocked after the DDL commit in this path:
```text
frontend response path
-> insertRecordToMoMysqlCompatibilityMode
-> background transaction / INSERT
-> MultiUpdate.updateFlushS3Info
-> DedupJoin / Merge
-> PipelineSignalReceiver
```
At the same time, the two remote CNs were blocked in remote execution / runtime-filter waiting. During cancellation they logged:
```text
connector cleanup timed out sending terminal Error signal
timeout=30s channel_len=14 channel_cap=14
pipeline_failed=true err=context canceled
```
The statement is not waiting on the original `CREATE DATABASE` DDL: `pkg/frontend/status_stmt.go` invokes `insertRecordToMoMysqlCompatibilityMode` after handling `*tree.CreateDatabase`, before writing the MySQL response. That helper opens a background transaction and inserts a row into `mo_catalog.mo_mysql_compatibility_mode`.
Confirmed source locations on main:
- `pkg/frontend/status_stmt.go`: post-`CREATE DATABASE` call to `insertRecordToMoMysqlCompatibilityMode`
- `pkg/frontend/authenticate.go`: `insertRecordToMoMysqlCompatibilityMode`, including its background `BEGIN` and metadata `INSERT`
### Hypothesis requiring owner confirmation
The evidence strongly indicates a distributed pipeline progress/lifecycle bug involving the internal compatibility-mode `INSERT`, `DedupJoin`/runtime-filter waiting, and terminal connector signaling. It is not yet proven which component first stops making progress, so this issue does not assign root cause to a specific function or recent commit.
## Impact
- Supported `CREATE DATABASE` can commit but appear to clients as a timed-out failure.
- A session becomes unusable after the timeout.
- Generic regression runs spend two minutes per affected connection and cannot complete under ordinary CI job limits.
- Retrying after an apparent failure is unsafe without first checking whether the database already exists.
## Regression coverage
After a product fix, add a multi-CN + Proxy regression (MOTR scenario driver or distributed BVT) that:
1. repeatedly creates fresh databases through Proxy and verifies each client receives OK;
2. validates the catalog row and database existence after every response;
3. exercises the internal metadata-insert path on each CN or controls routing explicitly;
4. verifies no stuck goroutine/pipeline, connector terminal-signal timeout, or unusable connection;
5. includes a bounded timeout so a recurrence fails quickly instead of consuming 120 seconds per case.
## Related
- [#27115](https://github.com/matrixorigin/matrixone/issues/27115) is related remote-pipeline terminal-error propagation work, but it covered duplicate-key errors being reported incorrectly and is closed. This report is a new `CREATE DATABASE` post-commit hang/timeout path.
- [#27876](https://github.com/matrixorigin/matrixone/issues/27876) also recorded connector terminal-error delivery problems after a COS upload failure, but it is a separate closed large-data/fileservice trigger.
- [#27575](https://github.com/matrixorigin/matrixone/issues/27575) is an open long `DROP DATABASE` issue after high-concurrency table churn; it has a different trigger and execution path.
Contributor guide
Research direction
Reproduce repeated CREATE DATABASE requests through Proxy in the stated 3-CN topology, then read the post-DDL path in pkg/frontend/status_stmt.go and insertRecordToMoMysqlCompatibilityMode in pkg/frontend/authenticate.go. Trace the background metadata INSERT alongside DedupJoin, PipelineSignalReceiver, and connector terminal signaling. Done means a multi-CN regression confirms bounded OK or clear errors, catalog consistency, and no stuck pipeline or unusable connection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100