cockroachdb / cockroachdb/cockroach

goschedstats: reinvestigate values of runnable goroutines that indicate overload

Open
#132,694 1 comment 0 reactions 0 assignees View on GitHub
A-admission-control C-investigation T-admission-control
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Back when we added this metrics as way of determining overload, in practice we would see values 10 or even more in non-overloaded clusters. We only saw degradation once this metric exceeded 30-50 or so.

Recently there was an observation that a customer was looking at this metric and it was seeing values of only 2 on overloaded clusters. I checked a running DRT cluster that was running TPCC at 100% CPU usage (with fairly high query latencies), and this value was between 2 and 6 (which I found very surprising).

It's possible there was some change inside the Go scheduler that changes when goroutines become runnable. We should investigate if there is a difference here between recent releases. If we find a difference, we may need to update the values we use for admission control.

CC @sumeerbhola @aadityasondhi

Jira issue: CRDB-43227

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.