fluent / fluent/fluent-bit

FLB_THROTTLE return for outputs

Open
#4,293 3 comments 0 reactions 0 assignees View on GitHub
enhancement long-term
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

Fluent Bit's behavior when an output is throttled is not ideal.

If one of my outputs gets a throttling error, I only have 3 choices for what to tell the core pipeline:
* FLB_OK: the entire chunk was successfully sent/handled
* FLB_RETRY: retry the entire chunk up to the max user configured retries
* FLB_ERROR: something went very wrong, don't retry

If I return FLB_RETRY, then that specific chunk will be retried with exponential backoff. However, the pipeline will continue to send my output new chunks. Thus, from the POV of my destination, I am not slowing down and backing off at all, I am continuing to send new data.

Thus, I think we need an FLB_THROTTLE return which would be like a special case for FLB_RETRY. It would tell the core to freeze the output for some period of time, and not send any new chunks. Basically, the core should backoff the entire output instance.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.