matrixorigin / matrixorigin/matrixone

[Bug]: CREATE DATABASE commits but hangs before responding on one CN in a multi-CN cluster

Open
#29,068 0 comments 0 reactions 0 assignees View on GitHub
kind/bug needs-triage severity/s-1
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

在 3 CN + Proxy 的正常 TKE 拓扑中,`CREATE DATABASE` 的 DDL 已在 TN 成功提交后,CN 在返回 MySQL OK 包之前执行内部 metadata 写入时卡住。客户端约 120 秒后收到:

```text
ERROR 2013 (HY000): Lost connection to MySQL server during query (timed out)
```

这不是 DDL 本身执行失败:目标数据库已经存在。故障会使客户端认为创建失败、连接不可用,并让后续大量无关的测试 case 在各自第一条 `CREATE DATABASE` 上超时。

## Environment

- Branch: `main`(CI 在运行开始时解析的官方 main)
- MatrixOne commit: `370c310a994de258ee01e87c31062590a7022f34`
- Date: `2026-09-17`
- Deployment: TKE, 3 CN / 1 TN / Proxy, CN `14C / 55GiB`
- Namespace: `mo-search-commit-370c310a99-20260917`
- Affected coordinator CN: `nightly-regression-dis-tp-cn-7wqk8`
- Test: [mo-nightly-regression run 35208053354 / MOTR job 105160533255](https://github.com/matrixorigin/mo-nightly-regression/actions/runs/35208053354/job/105160533255)
- MOTR commit: `4a1b1b40667a1fdbcef33eb2ed1c6916dfd2b8bf`

The latest `main` has advanced after this CI run; this issue deliberately records the exact official-main commit that failed in CI, rather than claiming an unexecuted later commit was verified.

## Steps to reproduce

1. Deploy a normal multi-CN cluster with Proxy (the observed topology has 3 CN and 1 TN). Do not inject node, network, memory, or storage failures.
2. Connect through Proxy using the MySQL protocol.
3. Repeatedly create a fresh database. Use a different name for each attempt:

```sql
CREATE DATABASE repro_create_db_001;
CREATE DATABASE repro_create_db_002;
CREATE DATABASE repro_create_db_003;
-- continue with new names
```

4. In the affected run, connections routed to one CN consistently hung. A representative statement was:

```sql
CREATE DATABASE motr_issue_27259;
```

5. After the client receives `ERROR 2013`, reconnect and inspect the database/catalog state:

```sql
SHOW DATABASES LIKE 'motr_issue_27259';
SELECT *
FROM mo_catalog.mo_mysql_compatibility_mode
WHERE dat_name = 'motr_issue_27259';
```

## Actual behavior

The following server/client timeline is confirmed for `CREATE DATABASE motr_issue_27259` (UTC):

1. `10:14:46.501`: connection was accepted by CN `nightly-regression-dis-tp-cn-7wqk8`.
2. `10:14:46.684`: TN received `tn.handle.create.database`.
3. `10:14:46.687`: TN committed the DDL; the TN-side DDL latency was about `3.185ms`.
4. The client did not receive an OK response.
5. `10:16:46`: the client-side 120-second query timeout expired and returned `ERROR 2013`.

There were 21 distinct cases with the same `Lost connection` failure in this one run. The log contains 42 text matches because each failure is printed once in the case summary and once as `FIRST FAILURE`.

In the exclusive phase the failure pattern was highly regular: one roughly every three new connections failed after ~120 seconds, while the two intervening cases completed in roughly 0.3–0.7 seconds. For example:

```text
g-04 -> timeout after ~120s
g-05 -> success in ~0.6s
g-06 -> success in ~0.5s
g-07 -> timeout after ~120s
```

The same failure hit unrelated suites (`01_transaction`, `02_gc_checkpoint_flush`, `04_snapshot_pitr`, and `14_issue_regression`) at their initial `CREATE DATABASE`; therefore these are not independent case or baseline failures.

On cancellation, the exclusive queue had completed only `62/197` cases, with `135` not started. Nineteen 120-second connection waits alone consumed about 38 minutes.

## Expected behavior

- Once `CREATE DATABASE` commits successfully, the client must receive an OK response in bounded time.
- Any post-DDL internal metadata failure must return a clear server error; it must not leave the client waiting until a MySQL connection timeout.
- A client-visible timeout must not occur after the DDL has already committed without an unambiguous outcome.
- One unhealthy remote pipeline must not turn independent new connections routed through Proxy into repeated 120-second hangs.

## Stability and controls

- Reproduction evidence: 21 matching client timeouts in one fresh 3-CN CI environment; the timeout/passing cadence correlates with Proxy routing among 3 CN replicas.
- Control: the two CN routing alternatives successfully completed the same initial `CREATE DATABASE` setup in sub-second time; TN committed both successful and failing-route DDL requests in milliseconds.
- Data correctness check: the failing statement had already committed its DDL at TN before the client timeout. The failure is response/lifecycle correctness, not a rejected DDL.
- Resource checks around the failures: no Pod restart increase, no OOMKilled, no panic/fatal evidence, low CPU usage (~0.2–0.43 cores of 14) and low CN memory (~2.1–2.6GiB of 55GiB). This excludes ordinary resource exhaustion as the trigger.

## Evidence from server profiles and logs

The affected coordinator CN was blocked after the DDL commit in this path:

```text
frontend response path
-> insertRecordToMoMysqlCompatibilityMode
-> background transaction / INSERT
-> MultiUpdate.updateFlushS3Info
-> DedupJoin / Merge
-> PipelineSignalReceiver
```

At the same time, the two remote CNs were blocked in remote execution / runtime-filter waiting. During cancellation they logged:

```text
connector cleanup timed out sending terminal Error signal
timeout=30s channel_len=14 channel_cap=14
pipeline_failed=true err=context canceled
```

The statement is not waiting on the original `CREATE DATABASE` DDL: `pkg/frontend/status_stmt.go` invokes `insertRecordToMoMysqlCompatibilityMode` after handling `*tree.CreateDatabase`, before writing the MySQL response. That helper opens a background transaction and inserts a row into `mo_catalog.mo_mysql_compatibility_mode`.

Confirmed source locations on main:

- `pkg/frontend/status_stmt.go`: post-`CREATE DATABASE` call to `insertRecordToMoMysqlCompatibilityMode`
- `pkg/frontend/authenticate.go`: `insertRecordToMoMysqlCompatibilityMode`, including its background `BEGIN` and metadata `INSERT`

### Hypothesis requiring owner confirmation

The evidence strongly indicates a distributed pipeline progress/lifecycle bug involving the internal compatibility-mode `INSERT`, `DedupJoin`/runtime-filter waiting, and terminal connector signaling. It is not yet proven which component first stops making progress, so this issue does not assign root cause to a specific function or recent commit.

## Impact

- Supported `CREATE DATABASE` can commit but appear to clients as a timed-out failure.
- A session becomes unusable after the timeout.
- Generic regression runs spend two minutes per affected connection and cannot complete under ordinary CI job limits.
- Retrying after an apparent failure is unsafe without first checking whether the database already exists.

## Regression coverage

After a product fix, add a multi-CN + Proxy regression (MOTR scenario driver or distributed BVT) that:

1. repeatedly creates fresh databases through Proxy and verifies each client receives OK;
2. validates the catalog row and database existence after every response;
3. exercises the internal metadata-insert path on each CN or controls routing explicitly;
4. verifies no stuck goroutine/pipeline, connector terminal-signal timeout, or unusable connection;
5. includes a bounded timeout so a recurrence fails quickly instead of consuming 120 seconds per case.

## Related

- [#27115](https://github.com/matrixorigin/matrixone/issues/27115) is related remote-pipeline terminal-error propagation work, but it covered duplicate-key errors being reported incorrectly and is closed. This report is a new `CREATE DATABASE` post-commit hang/timeout path.
- [#27876](https://github.com/matrixorigin/matrixone/issues/27876) also recorded connector terminal-error delivery problems after a COS upload failure, but it is a separate closed large-data/fileservice trigger.
- [#27575](https://github.com/matrixorigin/matrixone/issues/27575) is an open long `DROP DATABASE` issue after high-concurrency table churn; it has a different trigger and execution path.

Contributor guide

Open the contributing guide

Research direction

Reproduce repeated CREATE DATABASE requests through Proxy in the stated 3-CN topology, then read the post-DDL path in pkg/frontend/status_stmt.go and insertRecordToMoMysqlCompatibilityMode in pkg/frontend/authenticate.go. Trace the background metadata INSERT alongside DedupJoin, PipelineSignalReceiver, and connector terminal signaling. Done means a multi-CN regression confirms bounded OK or clear errors, catalog consistency, and no stuck pipeline or unusable connection.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.