fluent / fluent/fluent-bit

Add possibility to show repeat errors only once

Open
#11,641 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## **Is your feature request related to a problem? Please describe.**
If the same error occurs on many consecutive attempts to push data to an output, the error repeatedly shows up in the logs. This can drown out useful system logs, especially when system logs are also monitored by Fluent-bit.

For example, among others, we have a Systemd input and an InfluxDB output. If for some reason the InfluxDB endpoint is not reachable, an error with the HTTP status is generated in the log. This then gets picked up by the Systemd input and the InfluxDB output tries to output this new log as well, generating another error. Eventually this results in the `Storage.Backlog` filling up, generating even more errors for Systemd to log.

An example of the journalctl output after some time with InfluxDB down:
```
2026-03-23T13:07:57.940884Z - fluent-bit.service[269] [2026/03/23 13:07:57.940553400] [error] [input chunk] fail to drop enough chunks in order to place new data coming from input plugin systemd.0
2026-03-23T13:07:57.950398Z - fluent-bit.service[269] [2026/03/23 13:07:57.943235194] [error] [input chunk] no available chunk
2026-03-23T13:07:58.019220Z - fluent-bit.service[269] [2026/03/23 13:07:58.18009674] [error] [output:influxdb:influxdb.1] http_status=500
2026-03-23T13:07:58.019220Z - fluent-bit.service[269] {"code":"internal error","message":"unexpected error writing points to database: timeout"}
2026-03-23T13:07:58.020409Z - fluent-bit.service[269] [2026/03/23 13:07:58.18712622] [error] [output:influxdb:influxdb.1] http_status=500
2026-03-23T13:07:58.020409Z - fluent-bit.service[269] {"code":"internal error","message":"unexpected error writing points to database: timeout"}
```

In our case, the host is an embedded system which has a relatively small backlog buffer configured, with `Storage.Backlog.Mem_Limit 5M`.

## **Describe the solution you'd like**
A limit or timeout on consecutive equivalent error logs could solve this issue.

Basically: if InfluxDB is unreachable for a period, then comes back, then becomes unreachable again, both incidents should generate error logs, but not constantly.

The timeout would solve a scenario like e.g. if a series of 500 errors is interrupted once with a 503 error, the first and second blocks are really the same incident and should have the constraints applied. On the other hand, if the connection comes back for one attempt and drops again for the next, that would probably be a special case: is that the same incident or a new one?

## **Describe alternatives you've considered**
I could not find alternatives to this yet. Expanding the storage backlog is only a partial fix.

We considered creating our own patches, but so far these have been very rudimentary.

## **Additional context**
Version: 4.1.1

Contributor guide

Open the contributing guide

Research direction

Start at the InfluxDB output's repeated HTTP-status and error path, then trace how its retries and logging interact with the Systemd input and Storage.Backlog. Clarify equivalent-error and incident boundaries, including interruptions by different status codes, and define done as suppressing repeated logs while still reporting separate outages.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
observability-sre, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.