fluent / fluent/fluent-bit-go

[feature-request] Add event based processing for input plugins

Open
#59 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
202
Forks
61
Avg merge
7d 20h
Merged PRs (30d)
2

Description

Currently input plugins run on a schedule and are called once per second. This is not ideal for dealing with streaming / event driven data, as that data may reach the input plugin at an uneven rate. Events should ideally be collected as soon as they are available (I believe this is how other plugins work, like the `tail` plugin).

With the current implementation, the fastest rate data chunks could be consumed would be every second.

I believe this would require a wrapper of `flb_input_set_collector_event` in the plugin API, however from what I can tell, Fluent Bit core would probably need to be updated as well since the proxy plugin which is used to wrap Go plugins is responsible for setting this timer, i.e. by calling `flb_input_set_collector_time` (here: https://github.com/fluent/fluent-bit/blob/master/src/flb_plugin_proxy.c#L151-L155).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.