cockroachdb / cockroachdb/cockroach

roachtest: stage_failure failed

Open
#154,115 1 comment 0 reactions 0 assignees View on GitHub
branch-release-25.4 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.stage_failure [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20516654?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20516654?buildTab=artifacts#/backup-restore/mixed-version) on release-25.4 @ [9221c5318324b33455257088d3a48fae69346359](https://github.com/cockroachdb/cockroach/commits/9221c5318324b33455257088d3a48fae69346359):

```
test backup-restore/mixed-version failed: (mixedversion.go:862).func2: cluster.Stage: output:: TRANSIENT_ERROR(stage_failure): context canceled
(mixedversion.go:889).Run: mixed-version test failure while running step 8 (run "initialize tpcc workload"): full command output in run_080638.118751550_n5_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 4
test artifacts and logs in: /artifacts/backup-restore/mixed-version/run_1
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=true
- fs=ext4
- localSSD=true
- mvtDeploymentMode=system-only
- mvtVersions=v25.2.5 → v25.3.1 → release-25.4
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-20516654/backup-restore-mixed-version/1758787556243/1758787790015)

Same failure on other branches

- #151910 roachtest: stage_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #146357 roachtest: stage_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3]
- #146037 roachtest: stage_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.0-rc]
- #140132 roachtest: stage_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.1]
- #123276 roachtest: stage_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]
- #123193 roachtest: stage_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #123047 roachtest: stage_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*stage_failure.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-54737

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the failure in mixedversion.go around steps 8 and cluster.Stage, then inspect run_080638.118751550_n5_cockroach-workload-i.log under the linked artifacts. Use the Grafana trace and related issues to determine the cause of the context cancellation; done means identifying and addressing the recurring stage_failure or documenting why it is expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.