cockroachdb / cockroachdb/cockroach
release-26.2: roachtest: monitor_failure failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.monitor_failure [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414727?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414727?buildTab=artifacts#/schemachange/bulkingest/nodes=12/rows=4000000000/add_column) on release-26.2 @ [06d97da79c9dd587b631fc4171b855184609ce27](https://github.com/cockroachdb/cockroach/commits/06d97da79c9dd587b631fc4171b855184609ce27):
Failed with:
```
test schemachange/bulkingest/nodes=12/rows=4000000000/add_column failed: (monitor.go:267).Wait: monitor failure: context canceled
EOF [owner=test-eng]
(metric_utils.go:206).CloseExporter: failed to export perf stats: context canceled
test artifacts and logs in: /artifacts/schemachange/bulkingest/nodes=12/rows=4000000000/add_column/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21414727-1783488870-198-n12cpu4-0001 | 34.24.2.162 | 10.142.2.42 |
| teamcity-21414727-1783488870-198-n12cpu4-0002 | 34.75.5.21 | 10.142.2.38 |
| teamcity-21414727-1783488870-198-n12cpu4-0003 | 35.243.187.58 | 10.142.2.36 |
| teamcity-21414727-1783488870-198-n12cpu4-0004 | 34.148.216.190 | 10.142.2.44 |
| teamcity-21414727-1783488870-198-n12cpu4-0005 | 34.139.235.30 | 10.142.2.43 |
| teamcity-21414727-1783488870-198-n12cpu4-0006 | 34.139.98.158 | 10.142.2.37 |
| teamcity-21414727-1783488870-198-n12cpu4-0007 | 34.73.171.11 | 10.142.2.40 |
| teamcity-21414727-1783488870-198-n12cpu4-0008 | 104.196.121.133 | 10.142.2.35 |
| teamcity-21414727-1783488870-198-n12cpu4-0009 | 34.74.3.166 | 10.142.2.39 |
| teamcity-21414727-1783488870-198-n12cpu4-0010 | 35.190.135.40 | 10.142.2.51 |
| teamcity-21414727-1783488870-198-n12cpu4-0011 | 34.139.106.83 | 10.142.2.50 |
| teamcity-21414727-1783488870-198-n12cpu4-0012 | 34.73.121.104 | 10.142.1.235 |
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=4
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21414727/schemachange-bulkingest-nodes-12-rows-4000000000-add-column/1783516759567/1783535570627)
/cc @cockroachlabs/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*monitor_failure.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-65596
Contributor guide
Research direction
Start with the roachtest README, the failure artifacts and Grafana link, then inspect monitor.go:267 and metric_utils.go:206 around the context cancellation. Reproduce the schemachange/bulkingest/nodes=12/rows=4000000000/add_column run if possible. Done means the cancellation cause is understood, the relevant behavior is corrected, and the test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100