matrixorigin / matrixorigin/matrixone

ci bvt test failed: dn flush does not return result

Open
#24,317 2 comments 0 reactions 1 assignee Claimed by @jiangxinmeng1 View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Summary
In `Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)`,
`git4data/branch/diff/diff_9.sql` intermittently hangs on a DN flush call and
cascades into "no such table" failures. attempt 2 rerun passes, so the run
ends up green, but attempt 1 shows a real MO-side stall.

- Run: https://github.com/matrixorigin/matrixone/actions/runs/25551304929
- Failed job (attempt 1): https://github.com/matrixorigin/matrixone/actions/runs/25551304929/job/74999655375
- PR: #24308 (branch `0508-fix-bitmap`, bitmap fix — likely orthogonal)
- Summary line: `TOTAL:89395, SUCCESS:88250, FAILED:9, IGNORED:1135, ABNORAML:1, SUCCESS RATE:99%`
- Case: `test/distributed/cases/git4data/branch/diff/diff_9.sql` (total:75, success:65, failed:9, abnoraml:1)

## What happened (timeline)

1. **10:58:00** — start `diff_9.sql`.
2. **row:83** `select mo_ctl('dn', 'flush', 'test_gc_diff.c2_tar');`
→ `MO does not return result in 300000 ms.`
3. pprof collection also fails:
```
The MO debug service can not be reached, the pprof operation was failed.
(server 127.0.0.1, debug port 12345)
```
i.e. both the SQL port and the debug port are unresponsive — the MO process
is effectively hung, not just the SQL slow.
4. Tester reconnects:
```
Failed to execute sql select mo_ctl('cn','synccommit',''),
cause: Communications link failure
The last packet ... 12,812 ms ago.
```
5. After reconnect, every subsequent statement in the case fails with
`no such table`:
- row:97/98 `data branch diff c2_tar against c2_src ...` → `no such table diff_9.c2_tar`
- row:100 `select count(*) from c2_tar where b != a` → `table "c2_tar" does not exist`
- row:102/103 `drop table c2_src / c2_tar` → `no such table diff_9.{c2_src,c2_tar}`
- row:115/117 `mo_ctl('dn','flush','test_gc_diff.{c3_tar,c3_src}')` → `no such table ...`
- row:149/151 `mo_ctl('dn','flush','test_gc_diff.{t1,t2}')` → `no such table ...`

So the hang on DN flush is followed by catalog visibility loss for
`test_gc_diff.*` in the reconnected session.

## Suspected areas

- DN flush path under docker-compose multi-CN (PESSIMISTIC): something
blocks the flush RPC long enough that even `/debug` port becomes
unreachable, suggesting a global stall (lockservice keeper, checkpoint
writer, or mpool backpressure) rather than a single-query slow path.
- After reconnect, `test_gc_diff.*` tables disappear from the session's
view — either the DN container actually restarted around the hang, or
catalog state was rolled back / desynchronised between CN and DN.

Unlikely to be introduced by PR #24308 (`0508-fix-bitmap`), which is a
bitmap fix orthogonal to DN flush.

## Reproduction
Flaky — attempt 2 of the same run passed.

## Environment
- Workflow: MatrixOne ALL CI → Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
- Commit: 4005973e0c72edfb4aa566be37fbc041e98d559b
- Time: 2026-05-08 10:58 UTC

## Asks
- Confirm whether this is a known flake on `diff_9.sql` / `test_gc_diff` dataset.
- If not, would be good to capture DN goroutine dump / checkpoint state next
time the hang repeats — pprof on the normal debug port failed here,
so we may need an alternative probe.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.