grafana / grafana/pyroscope

Add an instance-level inflight bytes limit to the distributor

Open
#5,437 1 comment 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

An application-level flag such as `-distributor.max-inflight-bytes` can protect distributors from being `OOMKilled` when segment writers experience slowdowns or outages.

Prior art:
- Mimir: [`-distributor.instance-limits.max-inflight-push-requests(-bytes)`](https://github.com/grafana/mimir/blob/885eae7283962bc820b5fdcbfae12c26accfeb6d/pkg/distributor/instance_limits.go#L27-L37), enforced at [distributor.go#L1443](https://github.com/grafana/mimir/blob/885eae7283962bc820b5fdcbfae12c26accfeb6d/pkg/distributor/distributor.go#L1443)
- Loki: [`-distributor.max-inflight-bytes`](https://github.com/grafana/loki/blob/8d2011726173097c479cd2d2a982013ac312f536/pkg/distributor/distributor.go#L133), enforced at [distributor.go#L661-L669](https://github.com/grafana/loki/blob/8d2011726173097c479cd2d2a982013ac312f536/pkg/distributor/distributor.go#L661-L669)

Should be opt-in and apply for both the regular (sync) and the async write path.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating Pyroscope's distributor implementation and its regular and async write paths. Compare the enforcement points cited in Mimir's distributor.go and Loki's distributor.go, then trace how request sizes and configuration flags are handled locally. Done means an opt-in max-inflight-bytes limit protects both write paths and has coverage for the limit behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.