cockroachdb / cockroachdb/cockroach

release-26.3: roachtest: monitor_failure failed

Open
#172,360 2 comments 0 reactions 0 assignees View on GitHub
branch-release-26.3 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.monitor_failure [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414745?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414745?buildTab=artifacts#/tpcc/mixed-headroom/n5cpu16) on release-26.3 @ [b96b2082fca995b6a78f8563ed7ba31a9f10bd54](https://github.com/cockroachdb/cockroach/commits/b96b2082fca995b6a78f8563ed7ba31a9f10bd54):

Failed with:

```
test tpcc/mixed-headroom/n5cpu16 failed: (test_monitor.go:35).func1: test monitor: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc/mixed-headroom/n5cpu16/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21414745-1783491255-213-n5cpu16-0001 | 34.73.175.213 | 10.142.0.78 |
| teamcity-21414745-1783491255-213-n5cpu16-0002 | 34.24.252.211 | 10.142.0.25 |
| teamcity-21414745-1783491255-213-n5cpu16-0003 | 35.196.0.31 | 10.142.0.91 |
| teamcity-21414745-1783491255-213-n5cpu16-0004 | 34.139.1.13 | 10.142.0.65 |
| teamcity-21414745-1783491255-213-n5cpu16-0005 | 35.231.251.46 | 10.142.0.199 |

Parameters:
- MetamorphicFilesystem=zfs
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=16
- diskCount=0
- encrypted=true
- fs=zfs
- localSSD=true
- mvtDeploymentMode=shared-process
- mvtVersions=v26.2.1 → v26.3.0-beta.3-dev-b96b2082fca995b6a78f8563ed7ba31a9f10bd54 (release-26.3)
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21414745/tpcc-mixed-headroom-n5cpu16/1783521550115/1783535043505)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21414745-1783491255-213-n5cpu16&from_ts=1783521550115&to_ts=1783535043505&live=false&storage=flex_tier)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*monitor_failure.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65593

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/test_monitor.go at the reported line and read the roachtest README. Inspect the linked artifacts and Grafana or Datadog logs for the tpcc/mixed-headroom/n5cpu16 run to determine why the test monitor returned EOF. Done means the failure cause is addressed and the roachtest no longer fails for this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.