cockroachdb / cockroachdb/cockroach

release-26.3: roachtest: monitor_failure failed

Open
#172,361 4 comments 0 reactions 0 assignees View on GitHub
branch-release-26.3 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.monitor_failure [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414745?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414745?buildTab=artifacts#/ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global) on release-26.3 @ [b96b2082fca995b6a78f8563ed7ba31a9f10bd54](https://github.com/cockroachdb/cockroach/commits/b96b2082fca995b6a78f8563ed7ba31a9f10bd54):

Failed with:

```
test ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global failed: (cluster.go:2816).Run: context canceled
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global/cpu_arch=fips/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21414745-1783491255-249-n4cpu32-0001 | 35.227.62.52 | 10.142.0.213 |
| teamcity-21414745-1783491255-249-n4cpu32-0002 | 34.73.138.251 | 10.142.0.166 |
| teamcity-21414745-1783491255-249-n4cpu32-0003 | 34.75.206.195 | 10.142.0.50 |
| teamcity-21414745-1783491255-249-n4cpu32-0004 | 34.23.6.89 | 10.142.0.67 |

Parameters:
- arch=fips
- cloud=gce
- coverageBuild=false
- cpu=32
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21414745/ycsb-c-nodes-3-cpu-32-mvcc-range-keys-global/1783533678693/1783535190714)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21414745-1783491255-249-n4cpu32&from_ts=1783533678693&to_ts=1783535190714&live=false&storage=flex_tier)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*monitor_failure.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65594

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/README.md and the monitor_failure report for the ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global run. Inspect the linked TeamCity artifacts, Grafana dashboard, and Datadog logs to determine why the monitor user task called t.Fatal(). Done means identifying the failure cause and validating the appropriate fix or test change with a passing roachtest run.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
cloud, databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.