matrixorigin / matrixorigin/matrixone

[CI] Isolate shared race UT clusters and attribute resource stalls

Open
#29,039 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Summary

The Ubuntu/x86 race UT suite has repeated embedded-cluster control-plane
timeouts and cascading failures across otherwise unrelated packages. The
current diagnostics expose container-level memory peaks, but do not identify
which process, cache, or wait path owns the cost. At least one shared-test
cleanup path can leave a CN in `Draining` after a timed-out state transition,
which then makes later tests fail their readiness checks.

This issue tracks the common test-infrastructure and fixture-lifecycle problem;
the individual test signatures remain linked below.

## Evidence

### PR #29031

- [UT run 35176331513](https://github.com/matrixorigin/matrixone/actions/runs/35176331513)
- [Failed job 105058855532](https://github.com/matrixorigin/matrixone/actions/runs/35176331513/job/105058855532)
- `pkg/tests/issues/TestIssue26640CrossAccountRestoreRebindsPrivileges` failed
with `Error 1105 ... connection reset by peer` while executing an account
restore.
- Before the failure, LogService emitted proposal/checker-state timeouts and
WAL appends of several seconds.

### PR #29032

- [UT run 35177731314](https://github.com/matrixorigin/matrixone/actions/runs/35177731314)
- [Failed job 105063157387](https://github.com/matrixorigin/matrixone/actions/runs/35177731314/job/105063157387)
- The job failed in `pkg/embed`, `pkg/tests/arrowload`, and `pkg/tests/dml`
with several `context deadline exceeded` errors.
- `TestPreparedAPRuntimeCacheWorkspaceTransition` reported:

```text
waiting for both base-cluster CNs: context deadline exceeded
admission-ready CNs=[0-cn-13002=Draining 1-cn-13002=Working]
```

- `TestGroupConcatNamedTimeZoneRemoteOwner` immediately before it failed while
changing a CN to `Draining`. Its cleanup defer that restores `Working` is
registered only after that state-change call returns successfully. If the
request times out after changing server state, the test exits without
registering the restoration, leaving shared state for the next test.
- The same run also showed LogService and TN/CN heartbeat timeouts. This issue
does not yet claim that every timeout has the same root cause.

## Problem

The current race UT runner combines shared embedded clusters, multiple service
processes, race instrumentation, package-level parallelism, and a single
container resource budget. A test-level deadline or partially completed control
plane operation can therefore affect later packages. Container peak memory and
the historical `memory.max` event count are insufficient to distinguish:

- compiler/linker memory from test-process memory;
- anonymous memory from file cache and kernel memory;
- genuine retained state from short-lived race/build peaks;
- CPU, I/O, or control-plane stalls from a product failure.

## Expected behavior

The race UT suite should isolate test-owned cluster state, restore or discard a
fixture after an uncertain control-plane transition, and report the resource
owner and wait condition when a test exceeds its deadline. A failure in one
embedded package should not silently poison the readiness state of the next
package.

## Proposed work

1. Audit every shared-cluster state transition and register cleanup before the
transition, with a safe invalidation path when the final state is unknown.
2. Add post-test assertions for CN/TN work state, active sessions, service
health, and cluster ownership before a shared fixture is reused.
3. Record low-overhead per-process RSS, cgroup `memory.stat`, `memory.events`,
CPU pressure/throttling, and I/O pressure in the existing UT heartbeat.
4. Correlate the resource samples with admission waits, LogService/WAL
latency, and active test names.
5. Re-evaluate package and cluster scheduling after attribution data is
available. Preserve test coverage and avoid masking failures with retries,
sleeps, skipped cases, or blanket timeout increases.
6. Add exact-head repeated controls for the affected tests and distinguish
test-fixture contamination, runner resource contention, and product
liveness defects.

## Related issues

- #28763 — race UT restore case intermittently hits deadline under CI load
- #29020 — Ubuntu/x86 UT context timeout in a shared embedded-cluster test
- #29033 — Ubuntu/x86 UT view-metadata upgrade fence control-plane timeout
- #28638 — `TestDataBranchDiffAsFile` cleanup timeout on Ubuntu/x86
- #28661 — Ubuntu UT hard timeout leaves engine tests active

## Scope

This issue is about test infrastructure, shared embedded-cluster lifecycle, and
resource attribution. It does not infer a product memory leak from the current
container peak alone, and it does not attribute the failures in #29031 or
#29032 to their PR changes without same-head controls.

Contributor guide

Open the contributing guide

Research direction

Start with the shared embedded-cluster tests in pkg/embed and pkg/tests, especially TestPreparedAPRuntimeCacheWorkspaceTransition and TestGroupConcatNamedTimeZoneRemoteOwner, then trace the existing UT heartbeat. Done means uncertain transitions cannot poison later tests, resource ownership and wait conditions are recorded, and repeated exact-head controls distinguish fixture contamination from runner contention or product defects.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go
Domain
ci-cd, observability, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.