cockroachdb / cockroachdb/cockroach
roachtest: pprof_collection failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.pprof_collection [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414653?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414653?buildTab=artifacts#/sysbench/oltp_read_only/nodes=3/cpu=32/conc=256) on master @ [964a7c766443d16ef23c53990b465892507c7cc4](https://github.com/cockroachdb/cockroach/commits/964a7c766443d16ef23c53990b465892507c7cc4):
Failed with:
```
test sysbench/oltp_read_only/nodes=3/cpu=32/conc=256 failed: (monitor.go:267).Wait: monitor failure: TRANSIENT_ERROR(pprof_collection): getting profile for n2: failed to get profile from https://34.148.221.185:26258/debug/pprof/mutex?seconds=30 after 1m0s: Get "https://34.148.221.185:26258/debug/pprof/mutex?seconds=30": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
test artifacts and logs in: /artifacts/sysbench/oltp_read_only/nodes=3/cpu=32/conc=256/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21414653-1783489444-253-n4cpu32-0001 | 34.74.41.210 | 10.142.0.34 |
| teamcity-21414653-1783489444-253-n4cpu32-0002 | 34.148.221.185 | 10.142.0.22 |
| teamcity-21414653-1783489444-253-n4cpu32-0003 | 34.24.206.201 | 10.142.0.29 |
| teamcity-21414653-1783489444-253-n4cpu32-0004 | 34.74.84.145 | 10.142.0.37 |
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=32
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21414653/sysbench-oltp-read-only-nodes-3-cpu-32-conc-256/1783535966450/1783537742390)
See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21414653-1783489444-253-n4cpu32&from_ts=1783535966450&to_ts=1783537742390&live=false&storage=flex_tier)
/cc @cockroachlabs/test-eng
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*pprof_collection.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-65600
Contributor guide
Research direction
Start with the roachtest README and the failure reported at monitor.go:267, then inspect the pprof_collection run artifacts and the /debug/pprof/mutex request details. Compare the referenced Grafana and Datadog data for the affected node. Done means identifying and addressing the timeout so the roachtest completes successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100