cockroachdb / cockroachdb/cockroach
roachtest: multitenant-upgrade failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.
roachtest.multitenant-upgrade [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20516527?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20516527?buildTab=artifacts#/multitenant-upgrade) on master @ [60f00610beea03f524c86fa641ba75877799705f](https://github.com/cockroachdb/cockroach/commits/60f00610beea03f524c86fa641ba75877799705f):
```
(mixedversion.go:889).Run: mixed-version test failure while running step 14 (allow upgrade to happen on system tenant by resetting ``preserve_downgrade_option``): pq: reset-setting: result is ambiguous: error=ba: DeleteRange [/Table/6/1/"cluster.preserve_downgrade_option",/Table/6/1/"cluster.preserve_downgrade_option"/PrefixEnd), EndTxn(commit) [/Table/6/1/"cluster.preserve_downgrade_option"], [txn: 3fb26547], [can-forward-ts] RPC error: grpc: error reading from server: read tcp 10.142.1.190:37424->10.142.1.182:26257: use of closed network connection [code 14/Unavailable] [propagate] (last error: TransactionAbortedError(ABORT_REASON_NEW_LEASE_PREVENTS_TXN): "sql txn" meta={id=3fb26547 key=/Table/6/1/"cluster.preserve_downgrade_option" iso=Serializable pri=0.00258344 epo=0 ts=1758808878.743048504,1 min=1758808874.006188527,0 seq=2} lock=true stat=PENDING rts=1758808874.006188527,0 wto=false gul=1758808874.506188527,0)
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
```
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- fs=ext4
- localSSD=true
- mvtDeploymentMode=system-only
- mvtVersions=v24.3.19 → v25.1.10 → v25.2.5 → v25.3.1 → master
- runtimeAssertionsBuild=true
- ssd=0
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-20516527/multitenant-upgrade/1758808093122/1758809112456)
Same failure on other branches
- #153150 roachtest: multitenant-upgrade failed [C-test-failure O-roachtest O-robot T-db-server branch-release-25.2.5-rc]
/cc @cockroachdb/server
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant-upgrade.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-54761
Contributor guide
Research direction
Start with pkg/cmd/roachtest/README.md and the failure at mixedversion.go:889, then inspect the TeamCity artifacts at /artifacts/multitenant-upgrade/run_1 and the linked Grafana run. Compare this assertion-enabled failure with the related issue #153150 and determine whether it reproduces without assertions. Done means the cause is identified and the multitenant-upgrade run has a documented fix or follow-up.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100