cockroachdb / cockroachdb/cockroach

release-26.2.5-rc: roachtest: jepsen workload completed, but analysis timed out failed

Open
#172,748 1 comment 0 reactions 0 assignees View on GitHub
branch-release-26.2.5-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.jepsen workload completed, but analysis timed out [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21435807?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21435807?buildTab=artifacts#/jepsen/g2/subcritical-skews) on release-26.2.5-rc @ [0996c85a8557204febcf99cef0ada1a5e147d0e0](https://github.com/cockroachdb/cockroach/commits/0996c85a8557204febcf99cef0ada1a5e147d0e0):

Failed with:

```
test jepsen/g2/subcritical-skews failed: (jepsen.go:458).runJepsen: TRANSIENT_ERROR(jepsen workload completed, but analysis timed out): timed out
test artifacts and logs in: /artifacts/jepsen/g2/subcritical-skews/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21435807-1784785189-90-n6cpu4-0001 | 35.237.101.4 | 10.142.2.55 |
| teamcity-21435807-1784785189-90-n6cpu4-0002 | 34.139.111.25 | 10.142.1.124 |
| teamcity-21435807-1784785189-90-n6cpu4-0003 | 34.24.97.65 | 10.142.2.7 |
| teamcity-21435807-1784785189-90-n6cpu4-0004 | 34.26.6.152 | 10.142.0.115 |
| teamcity-21435807-1784785189-90-n6cpu4-0005 | 35.231.193.141 | 10.142.2.42 |
| teamcity-21435807-1784785189-90-n6cpu4-0006 | 34.73.177.22 | 10.142.2.66 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- metamorphicLeases=epoch
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21435807/jepsen-g2-subcritical-skews/1784800331528/1784803299689)

Same failure on other branches

- #172119 release-26.3: roachtest: jepsen workload completed, but analysis timed out failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.3]
- #172089 release-26.2.3-rc: roachtest: jepsen workload completed, but analysis timed out failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.3-rc]
- #168217 release-26.2.0-rc: roachtest: jepsen workload completed, but analysis timed out failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.0-rc]
- #166195 release-26.2: roachtest: jepsen workload completed, but analysis timed out failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2]
- #166066 roachtest: jepsen workload completed, but analysis timed out failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*jepsen workload completed, but analysis timed out.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66011

Contributor guide

Open the contributing guide

Research direction

Start with the failed TeamCity log and artifacts for jepsen/g2/subcritical-skews, then read pkg/cmd/roachtest/README.md and the jepsen.go:458 failure location. Use the linked Grafana view to correlate the workload and analysis timeout. Done means identifying why analysis timed out and verifying the roachtest no longer reports this failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
distributed-systems, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.