matrixorigin / matrixorigin/matrixone
[CI] Isolate shared race UT clusters and attribute resource stalls
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Summary
The Ubuntu/x86 race UT suite has repeated embedded-cluster control-plane
timeouts and cascading failures across otherwise unrelated packages. The
current diagnostics expose container-level memory peaks, but do not identify
which process, cache, or wait path owns the cost. At least one shared-test
cleanup path can leave a CN in `Draining` after a timed-out state transition,
which then makes later tests fail their readiness checks.
This issue tracks the common test-infrastructure and fixture-lifecycle problem;
the individual test signatures remain linked below.
## Evidence
### PR #29031
- [UT run 35176331513](https://github.com/matrixorigin/matrixone/actions/runs/35176331513)
- [Failed job 105058855532](https://github.com/matrixorigin/matrixone/actions/runs/35176331513/job/105058855532)
- `pkg/tests/issues/TestIssue26640CrossAccountRestoreRebindsPrivileges` failed
with `Error 1105 ... connection reset by peer` while executing an account
restore.
- Before the failure, LogService emitted proposal/checker-state timeouts and
WAL appends of several seconds.
### PR #29032
- [UT run 35177731314](https://github.com/matrixorigin/matrixone/actions/runs/35177731314)
- [Failed job 105063157387](https://github.com/matrixorigin/matrixone/actions/runs/35177731314/job/105063157387)
- The job failed in `pkg/embed`, `pkg/tests/arrowload`, and `pkg/tests/dml`
with several `context deadline exceeded` errors.
- `TestPreparedAPRuntimeCacheWorkspaceTransition` reported:
```text
waiting for both base-cluster CNs: context deadline exceeded
admission-ready CNs=[0-cn-13002=Draining 1-cn-13002=Working]
```
- `TestGroupConcatNamedTimeZoneRemoteOwner` immediately before it failed while
changing a CN to `Draining`. Its cleanup defer that restores `Working` is
registered only after that state-change call returns successfully. If the
request times out after changing server state, the test exits without
registering the restoration, leaving shared state for the next test.
- The same run also showed LogService and TN/CN heartbeat timeouts. This issue
does not yet claim that every timeout has the same root cause.
## Problem
The current race UT runner combines shared embedded clusters, multiple service
processes, race instrumentation, package-level parallelism, and a single
container resource budget. A test-level deadline or partially completed control
plane operation can therefore affect later packages. Container peak memory and
the historical `memory.max` event count are insufficient to distinguish:
- compiler/linker memory from test-process memory;
- anonymous memory from file cache and kernel memory;
- genuine retained state from short-lived race/build peaks;
- CPU, I/O, or control-plane stalls from a product failure.
## Expected behavior
The race UT suite should isolate test-owned cluster state, restore or discard a
fixture after an uncertain control-plane transition, and report the resource
owner and wait condition when a test exceeds its deadline. A failure in one
embedded package should not silently poison the readiness state of the next
package.
## Proposed work
1. Audit every shared-cluster state transition and register cleanup before the
transition, with a safe invalidation path when the final state is unknown.
2. Add post-test assertions for CN/TN work state, active sessions, service
health, and cluster ownership before a shared fixture is reused.
3. Record low-overhead per-process RSS, cgroup `memory.stat`, `memory.events`,
CPU pressure/throttling, and I/O pressure in the existing UT heartbeat.
4. Correlate the resource samples with admission waits, LogService/WAL
latency, and active test names.
5. Re-evaluate package and cluster scheduling after attribution data is
available. Preserve test coverage and avoid masking failures with retries,
sleeps, skipped cases, or blanket timeout increases.
6. Add exact-head repeated controls for the affected tests and distinguish
test-fixture contamination, runner resource contention, and product
liveness defects.
## Related issues
- #28763 — race UT restore case intermittently hits deadline under CI load
- #29020 — Ubuntu/x86 UT context timeout in a shared embedded-cluster test
- #29033 — Ubuntu/x86 UT view-metadata upgrade fence control-plane timeout
- #28638 — `TestDataBranchDiffAsFile` cleanup timeout on Ubuntu/x86
- #28661 — Ubuntu UT hard timeout leaves engine tests active
## Scope
This issue is about test infrastructure, shared embedded-cluster lifecycle, and
resource attribution. It does not infer a product memory leak from the current
container peak alone, and it does not attribute the failures in #29031 or
#29032 to their PR changes without same-head controls.
Contributor guide
Research direction
Start with the shared embedded-cluster tests in pkg/embed and pkg/tests, especially TestPreparedAPRuntimeCacheWorkspaceTransition and TestGroupConcatNamedTimeZoneRemoteOwner, then trace the existing UT heartbeat. Done means uncertain transitions cannot poison later tests, resource ownership and wait conditions are recorded, and repeated exact-head controls distinguish fixture contamination from runner contention or product defects.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, go
- Domain
- ci-cd, observability, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100