matrixorigin / matrixorigin/matrixone

[Bug]: Race UT restore case intermittently hits deadline under CI load

Open
#28,763 4 comments 0 reactions 1 assignee Claimed by @XuPeng-SH View on GitHub
bug/ut kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] I have checked existing open and closed issues.

Related, but not confirmed duplicates:

- #28317 and #28742 track `TestIssue26640ClusterRestoreRebindsSubscriptionPrivileges` deadlocks reported as `Error 20701`. This report tracks an intermittent CI `context deadline exceeded` with a same-head fail/pass; the root cause may overlap and should be reconciled rather than assumed.
- #25997 records historical backup-checkpoint timeouts under CI load. It is closed and is not this restore test.

### Branch Name

`codex/issue-28665-group-concat-warning` (PR #28747)

### Commit ID

`a1f1b528438dd59cd5b95a5a3e59820288a79094`

### Other Environment Information

- Runner: Ubuntu/x86 race UT runner `amd64-mo-shanghai-8c16g-92mgx-runner-z2m7k`.
- UT inputs: `ut_parallel=6`, `ut_sharded=false`.
- The heartbeat diagnostics showed 17 processes and peak RSS around 17.18 GB while embedded-cluster race tests were active. Treat this as evidence of possible resource pressure, not proof of OOM or root cause.

### Actual Behavior

The unchanged test had different outcomes on the same PR head:

1. Run [34670518213, attempt 1](https://github.com/matrixorigin/matrixone/actions/runs/34670518213?attempt=1), UT job [103490916610](https://github.com/matrixorigin/matrixone/actions/runs/34670518213/job/103490916610): `TestIssue26640ClusterRestoreRebindsSubscriptionPrivileges` failed after 128.60 seconds with `context deadline exceeded` (test caller line 142, shared helper line 423). Logs around the test also contain TN/CN command-poll deadline errors, logservice heartbeat/propose timeouts, connection resets, and slow commits.
2. Same-head rerun, [attempt 2](https://github.com/matrixorigin/matrixone/actions/runs/34670518213?attempt=2), UT job [103501227797](https://github.com/matrixorigin/matrixone/actions/runs/34670518213/job/103501227797): that test passed, taking 304.84 seconds. The overall UT job still failed later in the separate `TestBackupProtectionCheckpointProtection` case after 32.61 seconds; its forced checkpoint returned `context deadline exceeded` after roughly 30 seconds.
3. As another control, the same restore test passed in [run 34670510073](https://github.com/matrixorigin/matrixone/actions/runs/34670510073/job/103490895027) in 85.81 seconds (a different PR head).

The fail-then-pass result on the unchanged head establishes intermittent behavior for the restore test. The separate checkpoint timeout is recorded as a related observation only; it does not establish that the checkpoint test itself is flaky.

### Expected Behavior

The CI race suite should complete this restore regression reliably. When it does not, the failure should identify whether the cause is a product lock/liveness defect, a resource-starved cluster, or a test budget/setup assumption. Do not mask it with retries or a blanket timeout increase.

### Steps to Reproduce

1. Run the Ubuntu/x86 race UT workflow with the PR head `a1f1b528438dd59cd5b95a5a3e59820288a79094` and the same runner topology.
2. Compare the exact test outcome, duration, service logs, admission waits, and resource diagnostics across repeated runs without changing the commit.
3. Correlate any timeout with the restore SQL/helper phase and inspect whether a lock cycle or resource starvation is present.

### Additional information

The run’s same-head retry did not pass overall. This issue is scoped to the observed intermittent restore-test timeout and its CI execution conditions; it does not claim that the separate backup checkpoint timeout has the same cause. If both signatures reduce to the open lock-order defect in #28317, consolidate the tracking accordingly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.