fluent / fluent/fluent-bit

Allow customizing the number of outputs

Open
#5,224 23 comments 4 reactions 1 assignee Claimed by @leonardo-albertovich View on GitHub
community-feedback feature-request long-term
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

**Is your feature request related to a problem? Please describe.**

There is currently a [hard limit](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_routes_mask.h#L40-L42) on the number of outputs that can be configured.

This limit is not documented at the moment, there is an [issue](https://github.com/fluent/fluent-bit/issues/4201) for that.

The current landing page of fluentbit has a statement on scalability stating: _"1pb Data Throughput Across Thousands Of Sources And Destinations Daily"_. And: _Dynamic Routing ... Distribute data to multiple destinations with a zero copy strategy_ I misinterpreted this to fit my use-case.

**Describe the solution you'd like**

A configuration that allows to change this limit.

**Describe alternatives you've considered**

Fluentd seems to support many more outputs, but I have not tried implementing it as the footprint seems too high.

Some kind of sharding in fluent-bit would allow to keep the number of outputs fixed, and keep the limit. This sharding could be managed by the fluent-operator.

**Additional context**

We have an on-premise multi-tenant Kubernetes cluster where an API provides a varying endpoint for bulk ingestion in OpenSearch. Each new tenant on the cluster gets a dedicated OpenSearch tenant. We link the tenant in the k8s cluster to the ingestion endpoint using an output plugin. This endpoint is not known in advance. As tenants are self-serve, the amount of outputs might reach in the thousands.

Through a cluster management API, we use the [fluent-operator](https://github.com/fluent/fluent-operator) to easily drop ClusterOutput documents to manage the binding from tenant-namespaces to OpenSearch endpoints.

I have not seen other solutions that are as elegant as the plugin system of fluent-bit, maybe there is an equivalent in fluentd, but overhead does matter in a shared cluster setting so that a minimal footprint is a desired quality.

Open question: Allowing the increase of outputs will have a memory impact and other compute side effects. I am leaning on the intuition that the low footprint of fluent-bit allows for more vertical scaling, but I do not know the internals to warrant if the assumption holds.

References:

* #4159 Reference on an issue that points to this current limitation.
* [My comment](https://github.com/fluent/fluent-bit/issues/4159#issuecomment-1083139419) for describing this use-case.
* Similar use-case of [namespace templated looping](https://github.com/fluent/fluent-bit/issues/4159#issuecomment-939038961) on inputs from @lmuhlha

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.