cockroachdb / cockroachdb/cockroach
release-26.2: roachtest: jepsen monotonic setup hit unstarted sqlliveness failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.jepsen monotonic setup hit unstarted sqlliveness [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21470055?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21470055?buildTab=artifacts#/jepsen/monotonic/majority-ring-subcritical-skews) on release-26.2 @ [4706d03a79a26906da3a9e5ff757d2239de83227](https://github.com/cockroachdb/cockroach/commits/4706d03a79a26906da3a9e5ff757d2239de83227):
Failed with:
```
test jepsen/monotonic/majority-ring-subcritical-skews failed: (jepsen.go:572).runJepsen: TRANSIENT_ERROR(jepsen monotonic setup hit unstarted sqlliveness): full command output in run_145249.347410133_n6_bash-e-c-cd-mntdata1.log: COMMAND_PROBLEM: exit status 254
test artifacts and logs in: /artifacts/jepsen/monotonic/majority-ring-subcritical-skews/cpu_arch=arm64/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21470055-1787035486-233-n6cpu4-0001 | | 10.91.97.80 |
| teamcity-21470055-1787035486-233-n6cpu4-0002 | | 10.91.97.196 |
| teamcity-21470055-1787035486-233-n6cpu4-0003 | | 10.91.96.3 |
| teamcity-21470055-1787035486-233-n6cpu4-0004 | | 10.91.98.7 |
| teamcity-21470055-1787035486-233-n6cpu4-0005 | | 10.91.97.89 |
| teamcity-21470055-1787035486-233-n6cpu4-0006 | | 10.91.96.191 |
Parameters:
- arch=arm64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=default
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21470055/jepsen-monotonic-majority-ring-subcritical-skews/1787064669298/1787065022228)
Same failure on other branches
- #172693 roachtest: jepsen monotonic setup hit unstarted sqlliveness failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
/cc @cockroachlabs/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*jepsen monotonic setup hit unstarted sqlliveness.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-66790
Contributor guide
Research direction
Start with the failure output and artifacts for jepsen/monotonic/majority-ring-subcritical-skews, especially the referenced run_145249.347410133_n6_bash-e-c-cd-mntdata1.log, then read pkg/cmd/roachtest/README.md for the investigation workflow. Determine why setup encounters an unstarted sqlliveness state and verify the fix by rerunning the affected roachtest without this failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100