matrixorigin / matrixorigin/matrixone

[Bug]: coverage-enabled pessimistic BVT exceeds its 75-minute CI job budget

Open
#27,071 13 comments 0 reactions 1 assignee Claimed by @gouhongshen View on GitHub
deferred kind/bug
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] I searched open and closed `matrixorigin/matrixone` issues for the cancelled pessimistic BVT, the 75-minute timeout, the cancelled coverage producer, and the final snapshot case; no matching open issue was found. #26569 is not a duplicate: it covers coverage failures from a *superseded* workflow, whereas this run's active producer exhausted its own job timeout.

### Branch Name

`agent/issue-27039-8b73875c` (PR #27056 against `main`)

### Commit ID

`63bbdb6a5b97ff4dcdf2544ee98a277bbde4e313`

### Other Environment Information

- Hardware parameters: GitHub-hosted Ubuntu runner
- OS type: Ubuntu 22.04
- Others: `MatrixOne ALL CI`, `pull_request_target`, run 31620340116 attempt 1; job [Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)](https://github.com/matrixorigin/matrixone/actions/runs/31620340116/job/94194813961)
- Shared workflow revision: `matrixorigin/CI/.github/workflows/e2e-standalone-parallel.yaml@d703d5172ff67c3b37601bc998d81c19d35d9ccd`; the active coverage-enabled pessimistic job has `timeout-minutes: 75`.

### Actual Behavior

The active pessimistic BVT producer was cancelled by the job timeout before completing its assigned full BVT group. The job started at `2026-08-12T17:04:29Z` and was cancelled at `2026-08-12T18:19:53Z` (75 minutes 24 seconds). Its coverage build alone took from `17:05:34Z` to `17:54:36Z`; BVT then ran for about 25 minutes before Actions cancelled it while executing `snapshot/cluster_level_snapshot_restore_system_table_to_sys_account.sql`.

The dependent `Matrixone Utils CI / Coverage` job then failed its prerequisite guard without checking out or evaluating coverage because the pessimistic producer had been cancelled:

```text
prerequisites_ready: false
coverage merge skipped because at least one required producer was skipped or failed
```

This is not caused by PR #27056's clone regression. In the same cancelled pessimistic job, its new case completed before the timeout:

```text
clone_table_if_not_exists.sql ... total:56, success:56, failed:0, ignored:0, abnoraml:0
```

### Expected Behavior

The coverage-enabled pessimistic BVT producer should complete its assigned group and upload its coverage artifact, or the CI partitioning/job budget should be adjusted so a normal passing group cannot be cancelled solely by the 75-minute cap. A derived coverage merge failure should not obscure the producer timeout.

### Steps to Reproduce

1. Open or update a PR targeting `main` so `MatrixOne ALL CI` runs with the coverage-enabled `e2e-standalone-parallel.yaml` pessimistic producer.
2. Let the job execute `make GOBUILD_OPT=-cover build` and then the assigned pessimistic BVT group.
3. When the combined build plus BVT duration exceeds the job's 75-minute `timeout-minutes`, observe GitHub Actions cancel `Start BVT Test` before all scripts finish.
4. Observe the coverage merge receive `prerequisites_ready: false` and fail before coverage processing.

### Additional information

Evidence from the cancelled job log:

```text
2026-08-12T17:54:36Z Build MatrixOne completed
2026-08-12T17:54:50Z Start BVT Test started
2026-08-12T18:02:04Z clone_table_if_not_exists.sql ... total:56, success:56, failed:0
2026-08-12T18:19:35Z Start snapshot/cluster_level_snapshot_restore_system_table_to_sys_account.sql
2026-08-12T18:19:44Z The operation was canceled.
```

The cancellation therefore occurs after this PR's test has passed and while an unrelated snapshot test is in progress. The CI workflow's 75-minute cap is the confirmed root cause of this PR's red derived Coverage check. Evaluate whether the correct remedy is a larger budget, faster coverage build/BVT partitioning, or a separate handling of producer-timeout coverage reporting; preserve real producer failures.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.