fluent / fluent/fluent-bit

Add support for timeout-based Retry_Limit

Open
#10,369 8 comments 1 reaction 0 assignees View on GitHub
exempt-stale feature-request
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

**Is your feature request related to a problem? Please describe.**
Discussions link: https://github.com/fluent/fluent-bit/discussions/10360

Refer to https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configure-retries , there are 3 types of values for scheduler Retry_Limit: N, no_limits and no_retries.

For our case, the output is Loki and we set Retry_Limit with no_limites in order to not lost any logs when Loki is down.
In Loki site, we allow to accept these old logs within 2 days by setting proper value of ingester.max_chunk_age.
Now our scenario is that in a rainy case, we manually make Loki to offline and let fluent-bit to buffer/cache local chunk/logs more than 2 days, then online the Loki.
With the scheduler of retry, local chunks will be flushed to Loki one by one with each dedicated task_id.
However, for these timestamps that older than 48h, Loki will reject to accept them by reporting "write operation failed, older acceptable timestamp is xxxx" error. For fluent-bit, the flush action is not successful and will loop in the endless retries.

**Describe the solution you'd like**

Above is the actual issue for us, when Loki could be back within 48h, all local buffered chunks could be flushed to Loki with no_limits successfully, but when it is larger than 48h, some logs will be sent to Loki endless and Loki will always reject them.

Therefore, I would like to propose another values of Retry_Limit which is a timeout-based value.

For example, when Retry_Limit is set to 24h, then for one specific retry action will be same as no_limits from start, but when the retry action(maybe one task_id) reached 24h, it will stop the retry.

**Describe alternatives you've considered**

**Additional context**

Appreciate for any comments.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.