matrixorigin / matrixorigin/matrixone
[Bug]: Ubuntu/x86 race UT has flaky SQL Task store readiness failure
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Failure
The Ubuntu/x86 race UT job for PR #28636 failed in the exact test `TestIssue27719DropAccountCleansSQLTaskLifecycle` with a task-store readiness error. The failure is non-deterministic: the same test passed three consecutive times on both the PR head and clean current `up/main` with the CI-equivalent race/CGO configuration.
## CI evidence
- PR: https://github.com/matrixorigin/matrixone/pull/28636
- Exact head: `0323d8249677b9d46afb5c17f608562c1af1d1a4`
- Branch: `m-28583`
- Run attempt: `1`
- Run: https://github.com/matrixorigin/matrixone/actions/runs/34672696755
- Failed UT job: https://github.com/matrixorigin/matrixone/actions/runs/34672696755/job/103496985440
- Aggregate check: https://github.com/matrixorigin/matrixone/actions/runs/34672696755/job/103505015413
- CI command: `make ut UT_CONFIGURED=1 UT_PARALLEL=6 UT_TIMEOUT=40 UT_SHARD=all`
The complete UT log reports:
```text
github.com/matrixorigin/matrixone/pkg/tests/issues/isolated
TestIssue27719DropAccountCleansSQLTaskLifecycle
Error 20400 (HY000): invalid state task store not ready
issue_27719_test.go:322
exec failed: create task never_run schedule '0 0 0 1 1 *' timezone 'UTC' as begin select 1; end
FAIL_UT_CASES=TestIssue27719DropAccountCleansSQLTaskLifecycle,
make: *** [Makefile:513: ut] Error 1
```
The `CI Required` failure only propagates the UT result. Other product/build lanes passed. The run also ended artifact finalization with `ECONNRESET`, which is a secondary runner/network signal, not the primary test signature.
## Comparison evidence
The exact selected test was run with:
```text
GOWORK=off .agents/skills/mo-dev/scripts/mo-cgo-test -race -tags matrixone_test -short -v -json -p 1 -timeout 40m -count=1 -run '^TestIssue27719DropAccountCleansSQLTaskLifecycle$' ./pkg/tests/issues/isolated
```
- PR head `0323d8249677b9d46afb5c17f608562c1af1d1a4`: 3/3 passed, 41.19s, 42.71s, 40.89s.
- Clean official `up/main` `4f9d170a22deaabc2eee6b239a51a8cb40a39f9f`: 3/3 passed, 41.39s, 43.51s, 41.13s. The detached baseline native CGo inputs were built by `make cgo`.
- The PR diff changes prepared SIGN numeric-domain handling in planner/frontend code and adds the #28583 regression test. It does not modify `pkg/tests/issues/isolated/issue_27719_test.go`, the SQL Task store implementation, the embedded-cluster fixture, the UT runner, or the CI harness.
- The PR and clean-main pass logs can still contain short-lived `invalid state task store not ready` service-startup messages; the selected test reaches `PASS` and exits 0. This is consistent with a readiness race rather than a deterministic PR regression.
The open issue #28538 tracks a different global UT hard-timeout signature, and closed issue #27986 tracks an older readiness mechanism; neither covers this exact terminal test/error combination.
## Reproduction
1. Run the Ubuntu/x86 race UT workflow for the PR with the command above.
2. Select `pkg/tests/issues/isolated.TestIssue27719DropAccountCleansSQLTaskLifecycle`.
3. Observe whether the embedded SQL Task store is queried before it is ready; the failure surfaces as `Error 20400 (HY000): invalid state task store not ready` while creating `never_run`.
4. Repeat the exact race/CGO selection to observe that the same test can pass after startup timing changes.
## Expected behavior
Embedded-cluster SQL Task initialization and teardown should be deterministic, and a transient task-store readiness window should not become a terminal failure in an otherwise unrelated issue regression test.
Contributor guide
Assessment
This issue has not been assessed yet.