cockroachdb / cockroachdb/cockroach

release-26.2: roachtest: jepsen monotonic setup hit unstarted sqlliveness failed

Open
#173,473 2 comments 0 reactions 0 assignees View on GitHub
branch-release-26.2 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.jepsen monotonic setup hit unstarted sqlliveness [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21470055?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21470055?buildTab=artifacts#/jepsen/monotonic/majority-ring-subcritical-skews) on release-26.2 @ [4706d03a79a26906da3a9e5ff757d2239de83227](https://github.com/cockroachdb/cockroach/commits/4706d03a79a26906da3a9e5ff757d2239de83227):

Failed with:

```
test jepsen/monotonic/majority-ring-subcritical-skews failed: (jepsen.go:572).runJepsen: TRANSIENT_ERROR(jepsen monotonic setup hit unstarted sqlliveness): full command output in run_145249.347410133_n6_bash-e-c-cd-mntdata1.log: COMMAND_PROBLEM: exit status 254
test artifacts and logs in: /artifacts/jepsen/monotonic/majority-ring-subcritical-skews/cpu_arch=arm64/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21470055-1787035486-233-n6cpu4-0001 | | 10.91.97.80 |
| teamcity-21470055-1787035486-233-n6cpu4-0002 | | 10.91.97.196 |
| teamcity-21470055-1787035486-233-n6cpu4-0003 | | 10.91.96.3 |
| teamcity-21470055-1787035486-233-n6cpu4-0004 | | 10.91.98.7 |
| teamcity-21470055-1787035486-233-n6cpu4-0005 | | 10.91.97.89 |
| teamcity-21470055-1787035486-233-n6cpu4-0006 | | 10.91.96.191 |

Parameters:
- arch=arm64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=default
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21470055/jepsen-monotonic-majority-ring-subcritical-skews/1787064669298/1787065022228)

Same failure on other branches

- #172693 roachtest: jepsen monotonic setup hit unstarted sqlliveness failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*jepsen monotonic setup hit unstarted sqlliveness.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66790

Contributor guide

Open the contributing guide

Research direction

Start with the failure output and artifacts for jepsen/monotonic/majority-ring-subcritical-skews, especially the referenced run_145249.347410133_n6_bash-e-c-cd-mntdata1.log, then read pkg/cmd/roachtest/README.md for the investigation workflow. Determine why setup encounters an unstarted sqlliveness state and verify the fix by rerunning the affected roachtest without this failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.