cockroachdb / cockroachdb/cockroach
roachtest: jepsen monotonic setup hit unstarted sqlliveness failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.jepsen monotonic setup hit unstarted sqlliveness [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21431578?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21431578?buildTab=artifacts#/jepsen/monotonic/subcritical-skews) on master @ [0c646a426eedbf311fce246ada45adff0c2fb854](https://github.com/cockroachdb/cockroach/commits/0c646a426eedbf311fce246ada45adff0c2fb854):
Failed with:
```
test jepsen/monotonic/subcritical-skews failed: (jepsen.go:572).runJepsen: TRANSIENT_ERROR(jepsen monotonic setup hit unstarted sqlliveness): full command output in run_081117.307025996_n6_bash-e-c-cd-mntdata1.log: COMMAND_PROBLEM: exit status 254
test artifacts and logs in: /artifacts/jepsen/monotonic/subcritical-skews/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21431578-1784614012-68-n6cpu4-0001 | 20.102.115.127 | 10.1.0.124 |
| teamcity-21431578-1784614012-68-n6cpu4-0002 | 20.106.169.12 | 10.1.0.129 |
| teamcity-21431578-1784614012-68-n6cpu4-0003 | 57.151.107.245 | 10.1.0.13 |
| teamcity-21431578-1784614012-68-n6cpu4-0004 | 74.235.161.25 | 10.1.0.125 |
| teamcity-21431578-1784614012-68-n6cpu4-0005 | 40.76.230.234 | 10.1.0.128 |
| teamcity-21431578-1784614012-68-n6cpu4-0006 | 20.106.192.125 | 10.1.0.126 |
Parameters:
- arch=amd64
- cloud=azure
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=leader
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21431578/jepsen-monotonic-subcritical-skews/1784621200792/1784621696817)
See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21431578-1784614012-68-n6cpu4&from_ts=1784621200792&to_ts=1784621696817&live=false&storage=flex_tier)
/cc @cockroachlabs/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*jepsen monotonic setup hit unstarted sqlliveness.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-65954
Contributor guide
Research direction
Start with pkg/cmd/roachtest/README.md and the TeamCity artifacts for jepsen/monotonic/subcritical-skews, especially run_081117.307025996_n6_bash-e-c-cd-mntdata1.log. Trace the setup failure reported as an unstarted sqlliveness error and compare the relevant logs and metrics. Done means the cause is understood and the monotonic setup no longer fails for this condition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100