cockroachdb / cockroachdb/cockroach

roachtest: vm_host_error failed

Open
#174,632 1 comment 0 reactions 0 assignees View on GitHub
branch-release-25.4 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.vm_host_error [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21495730?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21495730?buildTab=artifacts#/tpccbench/nodes=3/cpu=16/write-optimization=buffering) on release-25.4 @ [ea8f26b12c6afe487a68577c7b99b06f02ec0f7b](https://github.com/cockroachdb/cockroach/commits/ea8f26b12c6afe487a68577c7b99b06f02ec0f7b):

```
test tpccbench/nodes=3/cpu=16/write-optimization=buffering failed: VMs received host error during the test run: teamcity-21495730-1788328005-477-n4cpu16-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2700).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/write-optimization=buffering/run_1/ssh/ssh_215522.049692718_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/write-optimization=buffering/run_1
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=16
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21495730/tpccbench-nodes-3-cpu-16-write-optimization-buffering/1788384863928/1788386567951)

Same failure on other branches

- #165414 roachtest: vm_host_error failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #134787 roachtest: vm_host_error failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3]
- #130411 roachtest: vm_host_error failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*vm_host_error.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67832

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the failure artifacts and SSH log linked in the report. Read cluster.go around line 2700 and inspect the tpccbench run to determine the VM host error and possible SSH failure cause. Done means the failure is understood and the recurring roachtest issue is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.