Add an instance-level inflight bytes limit to the distributor
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
An application-level flag such as `-distributor.max-inflight-bytes` can protect distributors from being `OOMKilled` when segment writers experience slowdowns or outages.
Prior art:
- Mimir: [`-distributor.instance-limits.max-inflight-push-requests(-bytes)`](https://github.com/grafana/mimir/blob/885eae7283962bc820b5fdcbfae12c26accfeb6d/pkg/distributor/instance_limits.go#L27-L37), enforced at [distributor.go#L1443](https://github.com/grafana/mimir/blob/885eae7283962bc820b5fdcbfae12c26accfeb6d/pkg/distributor/distributor.go#L1443)
- Loki: [`-distributor.max-inflight-bytes`](https://github.com/grafana/loki/blob/8d2011726173097c479cd2d2a982013ac312f536/pkg/distributor/distributor.go#L133), enforced at [distributor.go#L661-L669](https://github.com/grafana/loki/blob/8d2011726173097c479cd2d2a982013ac312f536/pkg/distributor/distributor.go#L661-L669)
Should be opt-in and apply for both the regular (sync) and the async write path.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating Pyroscope's distributor implementation and its regular and async write paths. Compare the enforcement points cited in Mimir's distributor.go and Loki's distributor.go, then trace how request sizes and configuration flags are handled locally. Done means an opt-in max-inflight-bytes limit protects both write paths and has coverage for the limit behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100