cockroachdb / cockroachdb/cockroach

release-26.2.3-rc: roachtest: jepsen workload completed, but analysis timed out failed

Open
#172,089 1 comment 0 reactions 0 assignees View on GitHub
branch-release-26.2.3-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.jepsen workload completed, but analysis timed out [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21404390?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21404390?buildTab=artifacts#/jepsen/g2/subcritical-skews) on release-26.2.3-rc @ [90d3b6080727dc810224a99903cb2e88b81e91ae](https://github.com/cockroachdb/cockroach/commits/90d3b6080727dc810224a99903cb2e88b81e91ae):

Failed with:

```
test jepsen/g2/subcritical-skews failed: (jepsen.go:458).runJepsen: TRANSIENT_ERROR(jepsen workload completed, but analysis timed out): timed out
test artifacts and logs in: /artifacts/jepsen/g2/subcritical-skews/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21404390-1782713363-105-n6cpu4-0001 | 34.74.73.152 | 10.142.3.8 |
| teamcity-21404390-1782713363-105-n6cpu4-0002 | 34.138.37.168 | 10.142.2.151 |
| teamcity-21404390-1782713363-105-n6cpu4-0003 | 104.196.14.231 | 10.142.2.158 |
| teamcity-21404390-1782713363-105-n6cpu4-0004 | 34.74.124.139 | 10.142.2.230 |
| teamcity-21404390-1782713363-105-n6cpu4-0005 | 34.148.110.189 | 10.142.2.224 |
| teamcity-21404390-1782713363-105-n6cpu4-0006 | 34.148.0.248 | 10.142.1.239 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- metamorphicLeases=default
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21404390/jepsen-g2-subcritical-skews/1782728156595/1782731094219)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*jepsen workload completed, but analysis timed out.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65227

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the jepsen/g2/subcritical-skews artifacts from TeamCity build 21404390. Review the failure logs and linked Grafana view to determine why analysis timed out after the workload completed. Done means the timeout cause is identified and the test or reporting behavior no longer produces this failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.