cockroachdb / cockroachdb/cockroach

roachtest: backup-restore/mixed-version failed

Open
#172,161 5 comments 0 reactions 0 assignees View on GitHub
A-disaster-recovery B-runtime-assertions-enabled branch-release-25.2 C-test-failure O-roachtest O-robot P-3 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.backup-restore/mixed-version [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21407470?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21407470?buildTab=artifacts#/backup-restore/mixed-version) on release-25.2 @ [6fd4e9a498f9d5b7d619de84aaba747d411622e0](https://github.com/cockroachdb/cockroach/commits/6fd4e9a498f9d5b7d619de84aaba747d411622e0):

```
(mixedversion.go:810).Run: mixed-version test failure while running step 86 (run "plan and run backups"): exhausted all 80 retries waiting for job 1188908846244397057 to finish, last err: job 1188908846244397057: current status "running", waiting for "succeeded"
test artifacts and logs in: /artifacts/backup-restore/mixed-version/run_1
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=true
- fs=ext4
- localSSD=true
- metamorphicBufferedSender=true
- mvtDeploymentMode=system-only
- mvtVersions=v24.1.29 → v24.2.10 → v24.3.33 → v25.1.10 → release-25.2
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21407470/backup-restore-mixed-version/1782885360255/1782901456384)

Same failure on other branches

- #172046 roachtest: backup-restore/mixed-version failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-25.4]
- #170322 roachtest: backup-restore/mixed-version failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-release-26.1]

/cc @cockroachlabs/disaster-recovery

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*backup-restore/mixed-version.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65306

Contributor guide

Open the contributing guide

Research direction

Start with the failure at mixedversion.go:810 and read the roachtest README to understand the backup-restore/mixed-version test. Inspect the artifacts and logs under /artifacts/backup-restore/mixed-version/run_1, then compare the Grafana run and the failures on the other branches. Done means identifying why the job remains running after 80 retries and addressing the test failure with evidence from a passing run.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cloud, databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.