influxdata / influxdata/telegraf

FR: introduce metric_buffer_size_limit

Open
#16,706 4 comments 0 reactions 0 assignees View on GitHub
feature request help wanted size/l
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

When running in K8S, telegraf is often running with a pod memory limit. Telegraf configuration should be done such way, that it wont trigger OOM. Likely, one of the biggest memory consumer is metrics buffer keeping metrics accumulated before flushes.

Current tunable `metric_buffer_limit` configures number of metrics which can be stored in the buffer. It is not obvious how to use that to keep telegraf memory under desired limit to prevent OOM: each metric has variable number of labels which in turn consume variable amount of memory. Currently `metric_buffer_limit` needs to be set empirically and final value heavily depends on type of metrics collected, which can change over time.

If telegraf could estimate total memory usage of all metrics in the buffer, then introducing `metric_buffer_size_limit` tunable would allow better control over memory utilization.

Contributor guide

Open the contributing guide

Research direction

Start by locating the implementation and tests for the existing metric_buffer_limit setting, then trace how buffered metrics are represented and sized. Define how metric_buffer_size_limit should estimate memory and how it interacts with the existing count limit; done means the behavior is documented and covered by tests for variable-sized metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.