cockroachdb / cockroachdb/cockroach

roachtest: multitenant-upgrade failed

Open
#154,148 2 comments 0 reactions 0 assignees View on GitHub
A-testing B-runtime-assertions-enabled branch-master C-test-failure O-roachtest O-robot sync-me
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.multitenant-upgrade [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20516527?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/20516527?buildTab=artifacts#/multitenant-upgrade) on master @ [60f00610beea03f524c86fa641ba75877799705f](https://github.com/cockroachdb/cockroach/commits/60f00610beea03f524c86fa641ba75877799705f):

```
(mixedversion.go:889).Run: mixed-version test failure while running step 14 (allow upgrade to happen on system tenant by resetting ``preserve_downgrade_option``): pq: reset-setting: result is ambiguous: error=ba: DeleteRange [/Table/6/1/"cluster.preserve_downgrade_option",/Table/6/1/"cluster.preserve_downgrade_option"/PrefixEnd), EndTxn(commit) [/Table/6/1/"cluster.preserve_downgrade_option"], [txn: 3fb26547], [can-forward-ts] RPC error: grpc: error reading from server: read tcp 10.142.1.190:37424->10.142.1.182:26257: use of closed network connection [code 14/Unavailable] [propagate] (last error: TransactionAbortedError(ABORT_REASON_NEW_LEASE_PREVENTS_TXN): "sql txn" meta={id=3fb26547 key=/Table/6/1/"cluster.preserve_downgrade_option" iso=Serializable pri=0.00258344 epo=0 ts=1758808878.743048504,1 min=1758808874.006188527,0 seq=2} lock=true stat=PENDING rts=1758808874.006188527,0 wto=false gul=1758808874.506188527,0)
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- fs=ext4
- localSSD=true
- mvtDeploymentMode=system-only
- mvtVersions=v24.3.19 → v25.1.10 → v25.2.5 → v25.3.1 → master
- runtimeAssertionsBuild=true
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-20516527/multitenant-upgrade/1758808093122/1758809112456)

Same failure on other branches

- #153150 roachtest: multitenant-upgrade failed [C-test-failure O-roachtest O-robot T-db-server branch-release-25.2.5-rc]

/cc @cockroachdb/server

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant-upgrade.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-54761

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/README.md and the failure at mixedversion.go:889, then inspect the TeamCity artifacts at /artifacts/multitenant-upgrade/run_1 and the linked Grafana run. Compare this assertion-enabled failure with the related issue #153150 and determine whether it reproduces without assertions. Done means the cause is identified and the multitenant-upgrade run has a documented fix or follow-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.