Add functionality to discard only failed chunks
- Dominant language
- Ruby
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 20
Description
From fluentd docs:
```
Fluentd will abort the attempt to transfer the failing chunks on the following conditions:
- The number of retries exceeds retry_max_times (default: none)
- The seconds elapsed since the first retry exceeds retry_timeout (default: 72h)
In these events, all chunks in the queue are discarded. If you want to avoid this, you can enable retry_forever to make Fluentd retry indefinitely.
```
So let's say you have a queue with 1000 chunks for example and if you use [fluent-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch) to dispatch the events to different elasticsearch intances - for example:
```
host elasticsearch-logging.${record['kubernetes']['namespace_name']}.svc
```
In this case if 1 chunks fails (exceeds the `retry_max_times` because its elastic instance was down) all the 1000 chunks in the queue will be discarded (999 will potentially succeed because you send them dynamically to different elastic hosts).
So isn't reasonable to add support to discard only failing chunks from the queue?
Contributor guide
Assessment
This issue has not been assessed yet.