fluent / fluent/fluent-bit

Support detailed memory usage metrics for input plugins (utilization of mem_buf_limit)

Open
#3,022 7 comments 5 reactions 1 assignee Claimed by @edsiper View on GitHub
enhancement
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

**Is your feature request related to a problem? Please describe.**

Its hard to figure out the ideal mem_buf_limit for your scenario. It'd be nice if the tail plugin or all input plugins could emit metrics on their current resource usage. That way you can know which input needs more memory, and right-size the limit.

Its not just about configuring the mem_buf_limit, this would also just be useful in general to profile Fluent Bit. Some sort of per-plugin memory usage stats.

**Describe the solution you'd like**

Metrics in prometheus endpoint/format for the memory usage and limit for the input plugins.

**Describe alternatives you've considered**

Trial and error, checking logs for over limit warnings, which is not ideal, since you only see an warning when you are over. Best case is to know when you are getting close to the limit.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.