fluent / fluent/fluentd

[Feature Request] provide option to throw_exception/block before buffer queue is full

Open
#3,681 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Is your feature request related to a problem? Please describe.

Due to some issues I have have been experiencing, I believe it would make sense to provide options to throw_exception/block before the buffer queue is full.

When preforming tests with file buffering I have noted at least these issues:
https://github.com/fluent/fluent-plugin-opensearch/issues/35
https://github.com/fluent/fluent-plugin-opensearch/issues/34

Even with these issues resolved, I believe providing options to throw_exception/block before the buffer queue is full makes sense.

### Describe the solution you'd like

I would suggest adding to the buffering and/or flushing parameters:
https://docs.fluentd.org/configuration/buffer-section#buffering-parameters
https://docs.fluentd.org/configuration/buffer-section#flushing-parameters

Options to set a high-water mark percentage, similar to chunk_full_threshold, and an action, similar to the overflow_action would be required.

For example:
chunk_highwater_action - throw_exception/block
chunk_highwater_threshold - percentage of chunk size threshold for executing chunk_highwater_action

### Describe alternatives you've considered

N/A

### Additional context

N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.