matrixorigin / matrixorigin/matrixone
ci bvt test failed: dn flush does not return result
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Summary
In `Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)`,
`git4data/branch/diff/diff_9.sql` intermittently hangs on a DN flush call and
cascades into "no such table" failures. attempt 2 rerun passes, so the run
ends up green, but attempt 1 shows a real MO-side stall.
- Run: https://github.com/matrixorigin/matrixone/actions/runs/25551304929
- Failed job (attempt 1): https://github.com/matrixorigin/matrixone/actions/runs/25551304929/job/74999655375
- PR: #24308 (branch `0508-fix-bitmap`, bitmap fix — likely orthogonal)
- Summary line: `TOTAL:89395, SUCCESS:88250, FAILED:9, IGNORED:1135, ABNORAML:1, SUCCESS RATE:99%`
- Case: `test/distributed/cases/git4data/branch/diff/diff_9.sql` (total:75, success:65, failed:9, abnoraml:1)
## What happened (timeline)
1. **10:58:00** — start `diff_9.sql`.
2. **row:83** `select mo_ctl('dn', 'flush', 'test_gc_diff.c2_tar');`
→ `MO does not return result in 300000 ms.`
3. pprof collection also fails:
```
The MO debug service can not be reached, the pprof operation was failed.
(server 127.0.0.1, debug port 12345)
```
i.e. both the SQL port and the debug port are unresponsive — the MO process
is effectively hung, not just the SQL slow.
4. Tester reconnects:
```
Failed to execute sql select mo_ctl('cn','synccommit',''),
cause: Communications link failure
The last packet ... 12,812 ms ago.
```
5. After reconnect, every subsequent statement in the case fails with
`no such table`:
- row:97/98 `data branch diff c2_tar against c2_src ...` → `no such table diff_9.c2_tar`
- row:100 `select count(*) from c2_tar where b != a` → `table "c2_tar" does not exist`
- row:102/103 `drop table c2_src / c2_tar` → `no such table diff_9.{c2_src,c2_tar}`
- row:115/117 `mo_ctl('dn','flush','test_gc_diff.{c3_tar,c3_src}')` → `no such table ...`
- row:149/151 `mo_ctl('dn','flush','test_gc_diff.{t1,t2}')` → `no such table ...`
So the hang on DN flush is followed by catalog visibility loss for
`test_gc_diff.*` in the reconnected session.
## Suspected areas
- DN flush path under docker-compose multi-CN (PESSIMISTIC): something
blocks the flush RPC long enough that even `/debug` port becomes
unreachable, suggesting a global stall (lockservice keeper, checkpoint
writer, or mpool backpressure) rather than a single-query slow path.
- After reconnect, `test_gc_diff.*` tables disappear from the session's
view — either the DN container actually restarted around the hang, or
catalog state was rolled back / desynchronised between CN and DN.
Unlikely to be introduced by PR #24308 (`0508-fix-bitmap`), which is a
bitmap fix orthogonal to DN flush.
## Reproduction
Flaky — attempt 2 of the same run passed.
## Environment
- Workflow: MatrixOne ALL CI → Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
- Commit: 4005973e0c72edfb4aa566be37fbc041e98d559b
- Time: 2026-05-08 10:58 UTC
## Asks
- Confirm whether this is a known flake on `diff_9.sql` / `test_gc_diff` dataset.
- If not, would be good to capture DN goroutine dump / checkpoint state next
time the hang repeats — pprof on the normal debug port failed here,
so we may need an alternative probe.
Contributor guide
Assessment
This issue has not been assessed yet.