matrixorigin / matrixorigin/matrixone

[Bug] mo_ctl dn flush timeout in git4data diff_7 under multi-CN pessimistic CI

Open
#24,630 0 comments 0 reactions 1 assignee Claimed by @jiangxinmeng1 View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

In PR CI, the multi-CN pessimistic compose BVT failed in `git4data/branch/diff/diff_7.sql`. The first real failure is that `mo_ctl("dn", "flush", "test.t1")` did not return within 300s. The following `no such table diff_7.t1` errors appear to be cascading failures after the timeout.

Job: https://github.com/matrixorigin/matrixone/actions/runs/26494748548/job/78020213206?pr=24573

## Environment

- Workflow job: `Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)`
- Result: failure
- Head SHA: `bcdaed95be1d129955bdbf55dcc897190eee7fe1`
- Test mode: multi-CN compose, pessimistic
- Failed case: `test/distributed/cases/git4data/branch/diff/diff_7.sql`

## Failure summary

Report summary:

```text
[SUMMARY] COST : 1722s, TOTAL :91601, SUCCESS : 90505, FAILED :6, IGNORED :1089, ABNORAML :1, SUCCESS RATE : 99%
```

Failed case summary:

```text
[/home/runner/work/matrixone/matrixone/test/distributed/cases/git4data/branch/diff/diff_7.sql] COST : 61.545s, TOTAL :33, SUCCESS :26, FAILED :6, IGNORED :0, ABNORAML :1, SUCCESS RATE : 78%
```

First failure from `report/error.txt`:

```text
[SCRIPT FILE]: /home/runner/work/matrixone/matrixone/test/distributed/cases/git4data/branch/diff/diff_7.sql
[ROW NUMBER]: 15
[SQL STATEMENT]: select mo_ctl("dn", "flush", "test.t1");
[EXPECT RESULT]:
➤ mo_ctl(dn, flush, test.t1)[12,0,0] 𝄀
{\n "method": "Flush",\n "result": [\n
[ACTUAL RESULT]:
MO does not return result in 300000 ms.
```

Cascading errors after the timeout:

```text
[ROW NUMBER]: 17
[SQL STATEMENT]: update t1 set b=3 where a = 1;
[ACTUAL RESULT]: no such table diff_7.t1

[ROW NUMBER]: 21
[SQL STATEMENT]: update t1 set b=4 where a = 1;
[ACTUAL RESULT]: no such table diff_7.t1

[ROW NUMBER]: 23
[SQL STATEMENT]: data branch diff t1 against t1{snapshot="sp"};
[ACTUAL RESULT]: no such table diff_7.t1

[ROW NUMBER]: 28
[SQL STATEMENT]: delete from t1 where a = 1;
[ACTUAL RESULT]: no such table diff_7.t1

[ROW NUMBER]: 30
[SQL STATEMENT]: data branch diff t1 against t1{snapshot="sp"};
[ACTUAL RESULT]: no such table diff_7.t1

[ROW NUMBER]: 32
[SQL STATEMENT]: drop table t1;
[ACTUAL RESULT]: no such table diff_7.t1
```

## Related SQL

The failing section of `diff_7.sql` is:

```sql
create table t1 (a int primary key, b int);
insert into t1 values(1,1);
insert into t1 values(2,2);

create snapshot sp for table test t1;

update t1 set b=2 where a = 1;
-- @ignore:0
select mo_ctl("dn", "flush", "test.t1");

update t1 set b=3 where a = 1;
-- @ignore:0
select mo_ctl("dn", "flush", "test.t1");

update t1 set b=4 where a = 1;

data branch diff t1 against t1{snapshot="sp"};
```

## Logs observed

From TN logs around this case:

```text
2026/05/27 06:40:45 +0000 tn.handle.create.database ... Name:diff_7
2026/05/27 06:41:46 +0000 tn.handle.create.relation ... 282480-diff_7:282485-t1
2026/05/27 06:41:46 +0000 tn.handle.drop.relation ... Id:282485 Name:t1 DatabaseName:diff_7
2026/05/27 06:41:46 +0000 tn.handle.drop.database ... Id:282480 Name:diff_7
```

The artifact also contains CN dump stacks. There are lockservice deadlock checker goroutines in the stack dumps, but I did not find a direct panic/OOM related to this failure. The main visible failure is the 300s no-return from `mo_ctl("dn", "flush", "test.t1")`.

## Expected behavior

`select mo_ctl("dn", "flush", "test.t1")` should return normally, and `data branch diff` should be able to continue using `test.t1`.

## Actual behavior

`select mo_ctl("dn", "flush", "test.t1")` did not return within 300s. After that, subsequent statements failed with `no such table diff_7.t1`.

## Current analysis

The primary issue seems to be a stuck or very slow DN flush triggered by `mo_ctl("dn", "flush", "test.t1")` in `diff_7.sql` under multi-CN pessimistic compose CI. The later `no such table diff_7.t1` errors look secondary because the case state was already broken after the flush timeout.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.