cockroachdb / cockroachdb/cockroach

release-26.3: roachtest: backup/ceph/reef/plain failed

Open
#172,385 5 comments 0 reactions 0 assignees View on GitHub
branch-release-26.3 C-test-failure O-roachtest O-robot T-field-eng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.backup/ceph/reef/plain [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21416295?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21416295?buildTab=artifacts#/backup/ceph/reef/plain) on release-26.3 @ [2fa3668a180eb255e627461824668261efe430b2](https://github.com/cockroachdb/cockroach/commits/2fa3668a180eb255e627461824668261efe430b2):

Failed with:

```
(cluster.go:2816).Run: full command output in run_062156.099343057_n4_sudo-snap-install-mi.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/backup/ceph/reef/plain/cpu_arch=arm64/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21416295-1783576955-20-n4cpu4-0001 | 34.23.50.196 | 10.142.0.112 |
| teamcity-21416295-1783576955-20-n4cpu4-0002 | 34.74.101.142 | 10.142.0.114 |
| teamcity-21416295-1783576955-20-n4cpu4-0003 | 35.237.238.201 | 10.142.1.97 |
| teamcity-21416295-1783576955-20-n4cpu4-0004 | 35.237.176.152 | 10.142.0.94 |

Parameters:
- arch=arm64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- metamorphicDRPC=true
- metamorphicLeases=default
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21416295/backup-ceph-reef-plain/1783578100532/1783579215942)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21416295-1783576955-20-n4cpu4&from_ts=1783578100532&to_ts=1783579215942&live=false&storage=flex_tier)

/cc @cockroachlabs/field-engineering

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*backup/ceph/reef/plain.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65621

Contributor guide

Open the contributing guide

Research direction

Start with run_062156.099343057_n4_sudo-snap-install-mi.log in the listed artifacts for backup/ceph/reef/plain, then read the roachtest README and inspect the linked Grafana and Datadog records. Identify why the ARM64 GCE run exits with status 1 and verify that the test passes again on release-26.3.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
cloud, databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.