cockroachdb / cockroachdb/cockroach

roachtest: pprof_collection failed

Open
#172,366 1 comment 0 reactions 0 assignees View on GitHub
branch-master O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.pprof_collection [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414653?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21414653?buildTab=artifacts#/sysbench/oltp_read_only/nodes=3/cpu=32/conc=256) on master @ [964a7c766443d16ef23c53990b465892507c7cc4](https://github.com/cockroachdb/cockroach/commits/964a7c766443d16ef23c53990b465892507c7cc4):

Failed with:

```
test sysbench/oltp_read_only/nodes=3/cpu=32/conc=256 failed: (monitor.go:267).Wait: monitor failure: TRANSIENT_ERROR(pprof_collection): getting profile for n2: failed to get profile from https://34.148.221.185:26258/debug/pprof/mutex?seconds=30 after 1m0s: Get "https://34.148.221.185:26258/debug/pprof/mutex?seconds=30": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
test artifacts and logs in: /artifacts/sysbench/oltp_read_only/nodes=3/cpu=32/conc=256/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21414653-1783489444-253-n4cpu32-0001 | 34.74.41.210 | 10.142.0.34 |
| teamcity-21414653-1783489444-253-n4cpu32-0002 | 34.148.221.185 | 10.142.0.22 |
| teamcity-21414653-1783489444-253-n4cpu32-0003 | 34.24.206.201 | 10.142.0.29 |
| teamcity-21414653-1783489444-253-n4cpu32-0004 | 34.74.84.145 | 10.142.0.37 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=32
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21414653/sysbench-oltp-read-only-nodes-3-cpu-32-conc-256/1783535966450/1783537742390)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21414653-1783489444-253-n4cpu32&from_ts=1783535966450&to_ts=1783537742390&live=false&storage=flex_tier)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*pprof_collection.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65600

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the failure reported at monitor.go:267, then inspect the pprof_collection run artifacts and the /debug/pprof/mutex request details. Compare the referenced Grafana and Datadog data for the affected node. Done means identifying and addressing the timeout so the roachtest completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.