cockroachdb / cockroachdb/cockroach

release-26.2: roachtest: vm_host_error failed

Open
#169,540 0 comments 0 reactions 0 assignees View on GitHub
branch-release-26.2 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.vm_host_error [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21334287?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21334287?buildTab=artifacts#/replicate/up/1to3) on release-26.2 @ [f4183be6fad9dc263decabf535abdf4b8c2116f4](https://github.com/cockroachdb/cockroach/commits/f4183be6fad9dc263decabf535abdf4b8c2116f4):

Failed with:

```
test replicate/up/1to3 failed: VMs received host error during the test run: teamcity-21334287-1777622077-65-n4cpu4-0002 (us-east1-c), teamcity-21334287-1777622077-65-n4cpu4-0002 (us-east1-c)

**Other Failures:**
(monitor.go:267).Wait: monitor failure: pq: replica unavailable: (n1,s1):1 unable to serve request to r84:/Table/113/2{-/7458496/5} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=7, sticky=1777641196.121835656,0]: lost quorum (down: (n2,s2):2); closed timestamp: 1777641808.770056373,0 (2026-05-01 13:23:28); raft status: {"id":"1","term":6,"vote":"1","commit":91,"lead":"0","leadEpoch":"0","raftState":"StatePreCandidate","applied":91,"progress":{},"leadtransferee":"0"}: encountered poisoned latch /Local/RangeID/84/r/AbortSpan/"b16ad8ae-7ba3-412e-a097-c1e8c5d17916"@0,0
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/replicate/up/1to3/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21334287-1777622077-65-n4cpu4-0001 | 34.26.19.247 | 10.142.1.25 |
| teamcity-21334287-1777622077-65-n4cpu4-0002 | 34.75.152.45 | 10.142.0.213 |
| teamcity-21334287-1777622077-65-n4cpu4-0003 | 35.231.168.117 | 10.142.0.196 |
| teamcity-21334287-1777622077-65-n4cpu4-0004 | 34.23.199.92 | 10.142.2.179 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=epoch
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21334287/replicate-up-1to3/1777637575262/1777642969027)

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*vm_host_error.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-63528

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.