matrixorigin / matrixorigin/matrixone
[Bug]: TestCheckLeaseFailed intermittently leaves ISCP watermark at 0-0
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Summary
The unsharded Ubuntu/x86 race UT intermittently fails `pkg/vm/engine/test.TestCheckLeaseFailed` because the ISCP watermark remains at `0-0` instead of reaching the expected HNSW job watermark. This is an untouched-subsystem flaky candidate, not a compile, type, static-analysis, security, or stable product-error failure.
## Source PR and failed checks
- PR: https://github.com/matrixorigin/matrixone/pull/28633
- Branch: `m-28618`
- Exact head: `d0d1b70daa982c8a8da53e4199629f91bd7f3edd`
- Run attempt 2: https://github.com/matrixorigin/matrixone/actions/runs/34490540322
- UT job: https://github.com/matrixorigin/matrixone/actions/runs/34490540322/job/103098611703
- Required-check job: https://github.com/matrixorigin/matrixone/actions/runs/34490540322/job/103111716025
- CI command: `make ut UT_CONFIGURED=1 UT_PARALLEL=6 UT_TIMEOUT=40 UT_SHARD=all`
## Failure signature
The complete failed-job log records:
```text
TestCheckLeaseFailed
change_handle_test.go:3210: Condition never satisfied
change_handle_test.go:3232: ISCP watermark did not reach expected state
account=0 table=16390 job=hnsw_idx expectation=reach 1789089214068965429-0 observed=true found=true current=0-0
--- FAIL: TestCheckLeaseFailed (22.21s)
```
The surrounding `rpc client closed` messages occur during teardown; the stable test failure is the unmet ISCP watermark assertion above.
## Independence evidence
Using the repository's `mo-cgo-test` wrapper with the CI race/tags/mode, the exact focused command was:
```text
.agents/skills/mo-dev/scripts/mo-cgo-test -race -tags matrixone_test -vet=off -short -count=1 -timeout=40m -p 1 -run '^TestCheckLeaseFailed$' ./pkg/vm/engine/test
```
- The exact PR head passed 3/3 independent runs.
- Clean detached `up/main` at `e7d46d6dfb27696211ca828c55e245d3bd1b634d` passed the same command 3/3.
- The PR changes only `pkg/frontend/authenticate.go`, `pkg/frontend/load_preplan_auth_test.go`, `pkg/frontend/mysql_cmd_executor.go`, and `pkg/tests/arrowload/arrow_load_lifecycle_test.go`. Neither the changed-file closure nor the test harness touches `pkg/vm/engine/test`, ISCP lease/watermark handling, HNSW propagation, or the embedded-cluster path used by this test.
- The other CI gates and the Arrow LOAD package lane passed; the failed UT job reported this case and an unrelated logservice race only.
## Reproduction
From a clean checkout, run the focused command above three times with the repository's CGo test wrapper. The CI failure was observed in the full unsharded command listed above under Ubuntu/x86 race execution.
The permitted failed-job rerun for run 34490540322 was already accepted and completed as attempt 2. Because this exact run has already consumed its one rerun, no additional rerun is requested here.
Contributor guide
Assessment
This issue has not been assessed yet.