matrixorigin / matrixorigin/matrixone
[Bug] mo_ctl dn flush timeout in git4data diff_7 under multi-CN pessimistic CI
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
In PR CI, the multi-CN pessimistic compose BVT failed in `git4data/branch/diff/diff_7.sql`. The first real failure is that `mo_ctl("dn", "flush", "test.t1")` did not return within 300s. The following `no such table diff_7.t1` errors appear to be cascading failures after the timeout.
Job: https://github.com/matrixorigin/matrixone/actions/runs/26494748548/job/78020213206?pr=24573
## Environment
- Workflow job: `Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)`
- Result: failure
- Head SHA: `bcdaed95be1d129955bdbf55dcc897190eee7fe1`
- Test mode: multi-CN compose, pessimistic
- Failed case: `test/distributed/cases/git4data/branch/diff/diff_7.sql`
## Failure summary
Report summary:
```text
[SUMMARY] COST : 1722s, TOTAL :91601, SUCCESS : 90505, FAILED :6, IGNORED :1089, ABNORAML :1, SUCCESS RATE : 99%
```
Failed case summary:
```text
[/home/runner/work/matrixone/matrixone/test/distributed/cases/git4data/branch/diff/diff_7.sql] COST : 61.545s, TOTAL :33, SUCCESS :26, FAILED :6, IGNORED :0, ABNORAML :1, SUCCESS RATE : 78%
```
First failure from `report/error.txt`:
```text
[SCRIPT FILE]: /home/runner/work/matrixone/matrixone/test/distributed/cases/git4data/branch/diff/diff_7.sql
[ROW NUMBER]: 15
[SQL STATEMENT]: select mo_ctl("dn", "flush", "test.t1");
[EXPECT RESULT]:
➤ mo_ctl(dn, flush, test.t1)[12,0,0] 𝄀
{\n "method": "Flush",\n "result": [\n
[ACTUAL RESULT]:
MO does not return result in 300000 ms.
```
Cascading errors after the timeout:
```text
[ROW NUMBER]: 17
[SQL STATEMENT]: update t1 set b=3 where a = 1;
[ACTUAL RESULT]: no such table diff_7.t1
[ROW NUMBER]: 21
[SQL STATEMENT]: update t1 set b=4 where a = 1;
[ACTUAL RESULT]: no such table diff_7.t1
[ROW NUMBER]: 23
[SQL STATEMENT]: data branch diff t1 against t1{snapshot="sp"};
[ACTUAL RESULT]: no such table diff_7.t1
[ROW NUMBER]: 28
[SQL STATEMENT]: delete from t1 where a = 1;
[ACTUAL RESULT]: no such table diff_7.t1
[ROW NUMBER]: 30
[SQL STATEMENT]: data branch diff t1 against t1{snapshot="sp"};
[ACTUAL RESULT]: no such table diff_7.t1
[ROW NUMBER]: 32
[SQL STATEMENT]: drop table t1;
[ACTUAL RESULT]: no such table diff_7.t1
```
## Related SQL
The failing section of `diff_7.sql` is:
```sql
create table t1 (a int primary key, b int);
insert into t1 values(1,1);
insert into t1 values(2,2);
create snapshot sp for table test t1;
update t1 set b=2 where a = 1;
-- @ignore:0
select mo_ctl("dn", "flush", "test.t1");
update t1 set b=3 where a = 1;
-- @ignore:0
select mo_ctl("dn", "flush", "test.t1");
update t1 set b=4 where a = 1;
data branch diff t1 against t1{snapshot="sp"};
```
## Logs observed
From TN logs around this case:
```text
2026/05/27 06:40:45 +0000 tn.handle.create.database ... Name:diff_7
2026/05/27 06:41:46 +0000 tn.handle.create.relation ... 282480-diff_7:282485-t1
2026/05/27 06:41:46 +0000 tn.handle.drop.relation ... Id:282485 Name:t1 DatabaseName:diff_7
2026/05/27 06:41:46 +0000 tn.handle.drop.database ... Id:282480 Name:diff_7
```
The artifact also contains CN dump stacks. There are lockservice deadlock checker goroutines in the stack dumps, but I did not find a direct panic/OOM related to this failure. The main visible failure is the 300s no-return from `mo_ctl("dn", "flush", "test.t1")`.
## Expected behavior
`select mo_ctl("dn", "flush", "test.t1")` should return normally, and `data branch diff` should be able to continue using `test.t1`.
## Actual behavior
`select mo_ctl("dn", "flush", "test.t1")` did not return within 300s. After that, subsequent statements failed with `no such table diff_7.t1`.
## Current analysis
The primary issue seems to be a stuck or very slow DN flush triggered by `mo_ctl("dn", "flush", "test.t1")` in `diff_7.sql` under multi-CN pessimistic compose CI. The later `no such table diff_7.t1` errors look secondary because the case state was already broken after the flush timeout.
Contributor guide
Assessment
This issue has not been assessed yet.