matrixorigin / matrixorigin/matrixone

[Bug]: Ubuntu/x86 UT view-metadata upgrade fence control-plane timeout

Open
#29,033 0 comments 0 reactions 0 assignees View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Summary

The Ubuntu/x86 unit-test lane for PR #28608 has a reproducible-once control-plane timing failure in the view-metadata upgrade overlap test. The PR's focused test and a clean official `up/main` control both pass repeatedly; the failure is not in the PR's window hash-partition change.

## Evidence

- PR/head: https://github.com/matrixorigin/matrixone/pull/28608 at `4f927ca4b2db8b93734bcc9bc6472b6106c928c3`
- Workflow run, attempt 2: https://github.com/matrixorigin/matrixone/actions/runs/35101322486/attempts/2
- Failed job: https://github.com/matrixorigin/matrixone/actions/runs/35101322486/job/105040747618 (`Matrixone CI execution / UT Test on Ubuntu/x86`, step `Unit Testing`)
- Required aggregate: https://github.com/matrixorigin/matrixone/actions/runs/35101322486/job/105055904835
- The same run's attempt 1 failed earlier during ONNX Runtime transport/download setup; the failed-job rerun was accepted once and produced this distinct attempt-2 failure.

The attempt-2 log shows:

```text
make ut UT_CONFIGURED=1 UT_PARALLEL=6 UT_TIMEOUT=40 UT_SHARD=all
pkg/tests/upgrade:TestV406UpgradeCreatesViewMetadataCatalogTables/catalog_fence_overlaps_uncommitted_upgrade
upgrade_v4_0_6_view_metadata_test.go:198:
catalog upgrade did not commit after releasing the overlap barrier
make: *** [Makefile:513: ut] Error 124
```

Immediately before the assertion, the embedded cluster log contains LogService/Dragonboat proposal and heartbeat timeouts, a WAL append taking about 6.4 seconds, and CN/TN schedule-command context deadlines. The recovered UT diagnostics show the runner reaching its 17,179,869,184-byte memory limit (peak 17,180,672,000 bytes) without an OOM kill; the suite then hit its 70-minute hard deadline and cancelled still-running tests.

## Attribution controls

- PR head: the exact focused test passed 3/3 with the repository `mo-cgo-test` wrapper, `-tags matrixone_test -short -race -count=3 -p 1 -timeout 40m`.
- Clean official `up/main` at `e4511d6af419354f76f413df7602d24bfffe5a54`: the same focused test and flags passed 3/3.
- The PR diff changes window hash-partition planning, session-variable cache invalidation, and related window tests/docs. The failed test exercises the view-metadata upgrade fence through `pkg/sql/compile`, `pkg/embed`, `pkg/cnservice`, `pkg/bootstrap`, and `pkg/catalog`; no changed hunk or changed symbol is on that fence path, and the test does not set `window_partition_algorithm`, prepare a window plan, or execute a window query.
- Other exact-head CI lanes, including shared build, static analysis, coverage UT, and BVT lanes, passed.

## Reproduction

1. Run the MatrixOne ALL CI Ubuntu/x86 UT job for the PR head.
2. Use `make ut UT_CONFIGURED=1 UT_PARALLEL=6 UT_TIMEOUT=40 UT_SHARD=all`.
3. Inspect the UT diagnostics around `pkg/tests/upgrade/upgrade_v4_0_6_view_metadata_test.go:198` and the preceding embedded-cluster control-plane timeouts.

This is recorded as an Ubuntu/x86 embedded-cluster control-plane / shared UT-budget flaky candidate. No product-code change is requested from PR #28608 based on the controls above.

Contributor guide

Open the contributing guide

Research direction

Start with pkg/tests/upgrade/upgrade_v4_0_6_view_metadata_test.go:198 and run `make ut UT_CONFIGURED=1 UT_PARALLEL=6 UT_TIMEOUT=40 UT_SHARD=all` on Ubuntu/x86. Inspect the embedded-cluster LogService, Dragonboat, WAL, CN/TN, and memory-budget diagnostics to establish whether the timeout is reproducible and identify its cause; done means documenting a confirmed cause and a scoped remediation or flaky-test finding.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.