cockroachdb / cockroachdb/cockroach
admission: integrate snapshot ingest with CPU limiter
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In an internal test cluster, when we had unbounded snapshot ingests, we discovered that `Store.HandleSnapshot` function showed high CPU usage and eventually led to an increased goroutine scheduler latency that ultimately led to spikes in SQL latency.
In an internal [thread](https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1714763200493659?thread_ts=1714588889.738319&cid=C01SRKWGHG8), we discussed that the ideal solution for this would be to use the elastic CPU limiter for such work since it was impacting the scheduler latency. Since this work is not technically elastic work, we would need to tweak the CPU limiter to also handle regular traffic and support higher than 1ms thresholds for pacing.
CPU profile attached. [cpuprof.2024-04-30T18_09_32.630.102.pprof.zip](https://github.com/cockroachdb/cockroach/files/15224414/cpuprof.2024-04-30T18_09_32.630.102.pprof.zip)
Some metric from when the overload happened can be found [here](https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1714671830742889?thread_ts=1714588889.738319&cid=C01SRKWGHG8).
Jira issue: CRDB-38467
Epic CRDB-58158
Contributor guide
Assessment
This issue has not been assessed yet.