cockroachdb / cockroachdb/cockroach

release-26.2: roachtest: monitor_failure failed

Open
#172,362 1 comment 0 reactions 0 assignees View on GitHub
branch-release-26.2 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.monitor_failure [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414727?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414727?buildTab=artifacts#/schemachange/bulkingest/nodes=12/rows=4000000000/add_column) on release-26.2 @ [06d97da79c9dd587b631fc4171b855184609ce27](https://github.com/cockroachdb/cockroach/commits/06d97da79c9dd587b631fc4171b855184609ce27):

Failed with:

```
test schemachange/bulkingest/nodes=12/rows=4000000000/add_column failed: (monitor.go:267).Wait: monitor failure: context canceled
EOF [owner=test-eng]
(metric_utils.go:206).CloseExporter: failed to export perf stats: context canceled
test artifacts and logs in: /artifacts/schemachange/bulkingest/nodes=12/rows=4000000000/add_column/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21414727-1783488870-198-n12cpu4-0001 | 34.24.2.162 | 10.142.2.42 |
| teamcity-21414727-1783488870-198-n12cpu4-0002 | 34.75.5.21 | 10.142.2.38 |
| teamcity-21414727-1783488870-198-n12cpu4-0003 | 35.243.187.58 | 10.142.2.36 |
| teamcity-21414727-1783488870-198-n12cpu4-0004 | 34.148.216.190 | 10.142.2.44 |
| teamcity-21414727-1783488870-198-n12cpu4-0005 | 34.139.235.30 | 10.142.2.43 |
| teamcity-21414727-1783488870-198-n12cpu4-0006 | 34.139.98.158 | 10.142.2.37 |
| teamcity-21414727-1783488870-198-n12cpu4-0007 | 34.73.171.11 | 10.142.2.40 |
| teamcity-21414727-1783488870-198-n12cpu4-0008 | 104.196.121.133 | 10.142.2.35 |
| teamcity-21414727-1783488870-198-n12cpu4-0009 | 34.74.3.166 | 10.142.2.39 |
| teamcity-21414727-1783488870-198-n12cpu4-0010 | 35.190.135.40 | 10.142.2.51 |
| teamcity-21414727-1783488870-198-n12cpu4-0011 | 34.139.106.83 | 10.142.2.50 |
| teamcity-21414727-1783488870-198-n12cpu4-0012 | 34.73.121.104 | 10.142.1.235 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=4
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21414727/schemachange-bulkingest-nodes-12-rows-4000000000-add-column/1783516759567/1783535570627)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*monitor_failure.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65596

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README, the failure artifacts and Grafana link, then inspect monitor.go:267 and metric_utils.go:206 around the context cancellation. Reproduce the schemachange/bulkingest/nodes=12/rows=4000000000/add_column run if possible. Done means the cancellation cause is understood, the relevant behavior is corrected, and the test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.