fluent / fluent/fluent-bit

Storage metric input_chunks.{plugin name}.status.overlimit only applies when storage.type is memory

Open
#6,712 1 comment 0 reactions 0 assignees View on GitHub
bug enhancement exempt-stale feature-request
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report/Improvement Needed/Poor User Experience

Currently we have a set of storage metrics: https://docs.fluentbit.io/manual/administration/monitoring#metric-descriptions

There is a metric `input_chunks.{plugin name}.status.overlimit` is set if we are over the `Mem_Buf_Limit`: https://github.com/fluent/fluent-bit/blob/master/src/flb_storage.c#L203

However, as noted here, `Mem_Buf_Limit` only applies when `storage.type memory` (the default) is set. Otherwise, the `storage.max_chunks_up` is the memory limiting setting: https://github.com/fluent/fluent-bit/issues/6663

This seems non-desirable and also potentially confusing for users. What metric can they use to check the overlimit status for filesystem storage enabled inputs?

Given that there are two entirely separate cases in the code here, I think it'd make sense if this metric is based on `storage.max_chunks_up` for the filesystem buffer case, and `Mem_Buf_Limit` for the memory only buffer case.

Is there any reason this wasn't implemented?

Was the current situation truly designed and desired, or just a side effect of how the code works?

Contributor guide

Open the contributing guide

Research direction

Start with the metric description in the monitoring documentation and the overlimit logic at src/flb_storage.c#L203, then read issue #6663 for the storage.max_chunks_up context. Trace how memory and filesystem buffering select their limits; done means the overlimit metric reflects the applicable limit for each storage type and its documented behavior is clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.