grafana / grafana/pyroscope

Use AIMD to tune max number of query goroutines

Open
#2,883 0 comments 2 reactions 0 assignees View on GitHub
backend performance storage
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

In many places in Pyroscope we use goroutine pools to parallelize querying the block store. In such cases, we make a SWAG and set a constant upper-bound. Here is an example of one such occurrence:

https://github.com/grafana/pyroscope/blob/bbcb729d463465c124e1661bc47a3ea9f2d5009b/pkg/phlaredb/block_querier.go#L1427-L1428

This works well to limit large queries from consuming far too many resources. However, it will also throttle large queries even when the deployment has the resources to handle a large query. If we could dynamically set the size of the goroutine pools, we could let large queries consume more resources when the resources are available, allowing quiet deployments to utilize its resources to the fullest extent.

One popular mechanism that enables this behavior is [AIMD](https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease). If we find a way to describe "congestion" of our system, we can implement AIMD to maximize resource usage when queries are made.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.