cockroachdb / cockroachdb/cockroach

release-26.2.5-rc: roachtest: backup/ceph/reef/plain failed

Open
#172,783 1 comment 0 reactions 0 assignees View on GitHub
B-runtime-assertions-enabled branch-release-26.2.5-rc C-test-failure O-roachtest O-robot T-field-eng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.backup/ceph/reef/plain [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21437715?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21437715?buildTab=artifacts#/backup/ceph/reef/plain) on release-26.2.5-rc @ [0996c85a8557204febcf99cef0ada1a5e147d0e0](https://github.com/cockroachdb/cockroach/commits/0996c85a8557204febcf99cef0ada1a5e147d0e0):

Failed with:

```
(cluster.go:2708).Run: full command output in run_105318.153384914_n4_sudo-radosgwadmin-ke.log: COMMAND_PROBLEM: exit status 139
test artifacts and logs in: /artifacts/backup/ceph/reef/plain/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21437715-1784889982-16-n4cpu4-0001 | 34.26.148.80 | 10.142.2.46 |
| teamcity-21437715-1784889982-16-n4cpu4-0002 | 34.23.222.44 | 10.142.2.36 |
| teamcity-21437715-1784889982-16-n4cpu4-0003 | 34.24.149.18 | 10.142.2.35 |
| teamcity-21437715-1784889982-16-n4cpu4-0004 | 34.23.18.13 | 10.142.2.215 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=true
- fs=ext4
- localSSD=true
- metamorphicLeases=epoch
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21437715/backup-ceph-reef-plain/1784890283698/1784890577350)

Same failure on other branches

- #172385 release-26.3: roachtest: backup/ceph/reef/plain failed [C-test-failure O-roachtest O-robot T-field-eng branch-release-26.3]
- #169620 release-26.2: roachtest: backup/ceph/reef/plain failed [C-test-failure O-roachtest O-robot T-field-eng branch-release-26.2]
- #169617 roachtest: backup/ceph/reef/plain failed [C-test-failure O-roachtest O-robot T-field-eng branch-master]
- #169614 roachtest: backup/ceph/reef/plain failed [C-test-failure O-roachtest O-robot T-field-eng branch-release-25.4]
- #169610 roachtest: backup/ceph/reef/plain failed [C-test-failure O-roachtest O-robot T-field-eng branch-release-26.1]
- #169605 roachtest: backup/ceph/reef/plain failed [C-test-failure O-roachtest O-robot T-field-eng branch-release-25.4.10-rc]
- #169602 roachtest: backup/ceph/reef/plain failed [C-test-failure O-roachtest O-robot T-field-eng branch-release-25.2]
- #169598 roachtest: backup/ceph/reef/plain failed [C-test-failure O-roachtest O-robot T-field-eng branch-release-24.3]

/cc @cockroachlabs/field-engineering

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*backup/ceph/reef/plain.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66050

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the linked TeamCity log, especially run_105318.153384914_n4_sudo-radosgwadmin-ke.log, then inspect the artifacts under /artifacts/backup/ceph/reef/plain/run_1. Trace the backup/ceph/reef/plain test and reproduce the exit-status-139 failure; done means the failure is explained and the test passes reliably on the affected branches.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.