matrixorigin / matrixorigin/matrixone

[Bug]: prepared AP runtime cache UT cleanup times out under WAL pressure

Open
#28,786 0 comments 0 reactions 1 assignee Claimed by @XuPeng-SH View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?
- [x] I checked the existing issues; no match was found.

### Branch Name
main

### Commit ID
2d79f8f833d170b899ce4d5748032545f65982ac (PR #28770 CI run)

### Other Environment Information
- CI run: https://github.com/matrixorigin/matrixone/actions/runs/34695875816
- Job: Matrixone CI execution / UT Test on Ubuntu/x86
- Failure occurred in `TestPreparedAPRuntimeCacheWorkspaceTransition/remote-only=true`.

### Actual Behavior
The UT failed during deferred cleanup at `pkg/tests/dml/prepared_cache_test.go:72` while calling `DebugUpdateCNWorkState` to restore the CN to `Working`. It returned `context deadline exceeded` / `internal error: debugUpdateCNWorkState`. Nearby logs show WAL append latency around 4.18 seconds, slow transaction commits, and missed logservice heartbeats. The sibling `remote-only=false` subtest passed.

### Expected Behavior
The test should restore its CN work state reliably under runner load, while keeping the draining-placement assertion and cleanup checks intact.

### Steps to Reproduce
1. Run the Linux/x86 UT workflow under high load.
2. Execute `TestPreparedAPRuntimeCacheWorkspaceTransition/remote-only=true`.
3. Observe whether the deferred restore completes after the test has put one CN into `Draining`.

### Additional information
This is one observed failure, not proof of a repeatable flake or a product defect. The failure is in cluster-state cleanup and coincides with WAL/heartbeat timeouts; PR #28770 does not modify prepared-plan caching or this test. Please investigate cleanup behavior and runner-pressure sensitivity without removing the placement or restoration assertions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.