matrixorigin / matrixorigin/matrixone

[Bug]: Unrelated CREATE/DROP and ALTER transactions deadlock on inverted View/SNAPSHOT catalog gates

Open
#28,317 3 comments 0 reactions 1 assignee Claimed by @LeftHandCold View on GitHub
kind/bug severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] I have checked existing open and closed issues, including issues created since 2026-09-05.

Related, but not currently demonstrated duplicates:

- #28181: ALTER DROP INDEX continues blocking after concurrent ODKU ends; same-table DML/DDL and persistent post-workload blocking, not the cross-database two-gate cycle demonstrated here.
- #28259: SNAPSHOT feature-registry initialization failure under UT coverage, not this lock cycle.
- #28079: CREATE ACCOUNT globally serializes on the View metadata gate; shares the View gate but reports queueing/timeouts rather than the DDL lock-order inversion here.
- #27825: RESTORE CLUSTER versus background lineage GC deadlock, different transactions/lock graph.

### Branch Name

main

### Commit ID

`9ae27d218dc952571ec2ce2b5575e2e61e844279`

### Other Environment Information

- Original old Main nightly MOTR job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/34041258710/job/101610206739
- Independent reproduction: Linux Docker standalone single CN, 4 CPU / 12 GiB container limit, loopback SQL endpoint. No MOTR, GitHub Runner, Proxy routing or TKE scheduling is required for the independent reproduction.
- Exact image: `ccr.ccs.tencentyun.com/matrixone-dev/matrixone@sha256:0e388a288b744485d4759e0e960145b725a2b9cfea61c67847f50a5d7d155015`
- `git_version()` returned `9ae27d218`.
- Two application connections use the system tenant but **different databases and different user tables**. Minimal fixture: three rows per table.

### Actual Behavior

An explicit CREATE/INSERT/DROP transaction in one database can deadlock with an ALTER transaction in another database, because MO implicitly acquires the global View metadata and SNAPSHOT lifecycle gates in opposite order.

This is not a report that every SQL deadlock is a bug. The detector correctly breaks a real cycle. The problem is the internal catalog lock-order inversion coupling otherwise unrelated user DDL transactions.

In the original nightly at `2026-09-07 03:17:08.932541 UTC`, MOTR TC-02 failed on `DROP TABLE ephemeral`, while issue26164's ALTER/index-copy path ran concurrently. Server logs reported:

```text
9ff325d4fbab783618d2c5153a0101c9
<= e9a0447544da72bb18d2c51521d2af1e
<= 9ff325d4fbab783618d2c5153a0101c9
```

The victim's internal statement was:

```sql
update mo_catalog.mo_feature_registry
set scope_spec=scope_spec, updated_at=updated_at
where feature_code='SNAPSHOT';
```

The peer was creating an ALTER copy table and accessing the View gate. We then reproduced independently to rule out MOTR/Family harness behavior as a necessary cause.

### Steps to Reproduce

Use a disposable instance of the exact image above. Use unused database names; do not run cleanup against existing user databases.

Preparation, autocommit:

```sql
CREATE DATABASE mo1543_ddl_order_a;
CREATE DATABASE mo1543_ddl_order_b;
CREATE TABLE mo1543_ddl_order_b.t(
id INT PRIMARY KEY, v INT, INDEX idx_v(v)
);
INSERT INTO mo1543_ddl_order_b.t VALUES(1,1),(2,2),(3,3);
```

Connection A (leave transaction open):

```sql
USE mo1543_ddl_order_a;
BEGIN;
CREATE TABLE ephemeral(id INT PRIMARY KEY);
INSERT INTO ephemeral VALUES(1),(2),(3);
```

Connection B, execute asynchronously while A remains open:

```sql
USE mo1543_ddl_order_b;
BEGIN;
ALTER TABLE t ADD COLUMN c1 INT NOT NULL DEFAULT 1;
```

Using an observer connection, inspect `SELECT * FROM mo_locks() l`. Wait until B is waiting for A's `mo_catalog/mo_view_refresh` gate, rather than relying on a long arbitrary sleep. The automated test polled with a bounded four-second deadline.

Then connection A:

```sql
DROP TABLE ephemeral;
COMMIT;
```

Observed in the bounded two-connection test:

```text
observed_wait_before_drop: true
DROP TABLE ephemeral: passed
ALTER TABLE t ADD COLUMN c1 ...: (20701, 'deadlock detected')
```

The victim selection is not part of the contract; either transaction can be selected. Roll back the failed transaction, finish/close both connections, then drop the two task-owned databases. No application-level accesses to SNAPSHOT or View metadata are needed to trigger this.

### Lock evidence and source analysis

In the independent reproduction at `2026-09-07 06:58:02.950 UTC`:

- A: `70f70f4866a9b62a18d2f72919316b76`
- B: `70f70f4866a9b62a18d2f72919316b7a`
- A held the `table_id=2` Exclusive point lock for `mo_catalog/mo_view_refresh`; its `lock_wait` contained B.
- B held the Exclusive SNAPSHOT point lock, local `table_id=272476`, `lock_content=4601534e415053484f5400`.
- The server reported `...b7a <= ...b76 <= ...b7a`.
- B's internal failing statement was the View gate SELECT FOR UPDATE during `CREATE TABLE mo1543_ddl_order_b.t_copy_`; `process/process.go:707` reported `Create copy table for alter table`.
- After transactions and test cleanup completed, `mo_locks()` returned zero rows.

The resulting graph is:

```text
A: CREATE retains View gate -> DROP requests SNAPSHOT gate
B: ALTER retains SNAPSHOT gate -> copy CREATE requests View gate
```

Exact-revision source:

- [view_metadata_recovery.go](https://github.com/matrixorigin/matrixone/blob/9ae27d218dc952571ec2ce2b5575e2e61e844279/pkg/sql/compile/view_metadata_recovery.go#L62): `viewMetadataRequireRevalidationSQL()` starts with `ViewMetadataLifecycleGateSQL`.
- [view_metadata.go](https://github.com/matrixorigin/matrixone/blob/9ae27d218dc952571ec2ce2b5575e2e61e844279/pkg/sql/compile/view_metadata.go): `viewMetadataRefreshAvailable()` still calls `requireViewMetadataRevalidationInTxn()` when lifecycle refresh is disabled. The statements run in the caller transaction; feature=false does not eliminate this lock.
- [catalog/view_metadata.go](https://github.com/matrixorigin/matrixone/blob/9ae27d218dc952571ec2ce2b5575e2e61e844279/pkg/catalog/view_metadata.go#L30): View gate selects the `mo_tables` row for `mo_catalog.mo_view_refresh FOR UPDATE`.
- [ddl.go](https://github.com/matrixorigin/matrixone/blob/9ae27d218dc952571ec2ce2b5575e2e61e844279/pkg/sql/compile/ddl.go): CREATE ends with `refreshViewsAfterRelationMutation`; persistent DROP enters `lockDataBranchLineageOwnerLifecycle` before its metadata work.
- [alter.go](https://github.com/matrixorigin/matrixone/blob/9ae27d218dc952571ec2ce2b5575e2e61e844279/pkg/sql/compile/alter.go#L1177): ALTER acquires the lineage owner gate before its copy-create path.
- [lineage_publication_lock.go](https://github.com/matrixorigin/matrixone/blob/9ae27d218dc952571ec2ce2b5575e2e61e844279/pkg/frontend/databranchutils/lineage_publication_lock.go): the owner lifecycle gate updates the SNAPSHOT feature-registry row.

### Expected Behavior

Supported explicit DDL transactions on unrelated databases should not be forced into an avoidable deadlock by inconsistent internal lifecycle-gate order. Preserve View revalidation and Snapshot/lineage atomicity, but use a consistent coordination protocol across the entire explicit transaction, not merely within one DDL statement.

Do not treat disabling parallel MOTR, changing 1TP+2AP topology, or blindly retrying all DDL in the runner as the product fix.

### Additional information

Controls on the same standalone image:

| Workload | Results |
|---|---|
| CREATE/INSERT/DROP only, serial + four workers | 50/50 passed |
| DROP/ROLLBACK only, serial + four workers | 50/50 passed |
| ALTER only, serial + four workers | 50/50 passed |
| Mixed DDL, serial | 10/10 passed |
| Mixed DDL, four workers in separate DBs | 8 explicit 20701 errors |

The mixed group also had two barrier-abort errors caused by the earlier failures; these are harness consequences, **not** additional product failures. Total: 175 attempts, 165 passed. All 20 fixture databases cleaned successfully. No OOM or container restart occurred.

The additional two-connection/three-row ordered reproduction above completed once and captured the exact lock relationship. We do not claim an unmeasured repeat rate. Its diagnostic Python process returned zero because it caught and printed the SQL failure; zero is not a SQL PASS verdict.

Impact: normal DDL transactions fail and must be retried despite disjoint user objects. No data corruption was demonstrated. First introducing commit and complete affected-version range are not yet established; the tested revision is not a claim about every later main revision. This is separate from large PK-update retained-lock-budget/coarsening deadlocks.

Suggested regression: two distinct databases, bounded phase synchronization using the actual lock wait, CREATE/INSERT/DROP versus ALTER-copy, successful final schema/data checks, rollback/cancellation cleanup, then multi-CN coverage. Three rows suffice; no big-data prerequisite.

Raw local evidence is retained under `/tmp/mo1543-evidence/ddl-repro-20260907/` (not a public download link). Key SQL, lock IDs, results and public nightly/source links are embedded above so the report does not depend on access to that workstation. The disposable container has been removed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.