cockroachdb / cockroachdb/cockroach
release-26.3: roachtest: monitor_failure failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.monitor_failure [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414745?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414745?buildTab=artifacts#/ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global) on release-26.3 @ [b96b2082fca995b6a78f8563ed7ba31a9f10bd54](https://github.com/cockroachdb/cockroach/commits/b96b2082fca995b6a78f8563ed7ba31a9f10bd54):
Failed with:
```
test ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global failed: (cluster.go:2816).Run: context canceled
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global/cpu_arch=fips/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21414745-1783491255-249-n4cpu32-0001 | 35.227.62.52 | 10.142.0.213 |
| teamcity-21414745-1783491255-249-n4cpu32-0002 | 34.73.138.251 | 10.142.0.166 |
| teamcity-21414745-1783491255-249-n4cpu32-0003 | 34.75.206.195 | 10.142.0.50 |
| teamcity-21414745-1783491255-249-n4cpu32-0004 | 34.23.6.89 | 10.142.0.67 |
Parameters:
- arch=fips
- cloud=gce
- coverageBuild=false
- cpu=32
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21414745/ycsb-c-nodes-3-cpu-32-mvcc-range-keys-global/1783533678693/1783535190714)
See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21414745-1783491255-249-n4cpu32&from_ts=1783533678693&to_ts=1783535190714&live=false&storage=flex_tier)
/cc @cockroachlabs/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*monitor_failure.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-65594
Contributor guide
Research direction
Start with pkg/cmd/roachtest/README.md and the monitor_failure report for the ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global run. Inspect the linked TeamCity artifacts, Grafana dashboard, and Datadog logs to determine why the monitor user task called t.Fatal(). Done means identifying the failure cause and validating the appropriate fix or test change with a passing roachtest run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- cloud, databases, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100