fluent / fluent/fluent-bit

Allow output plugins to configure a max chunk size

Open
#1,938 28 comments 32 reactions 0 assignees View on GitHub
AWS community-feedback feature-request long-term
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

@edsiper and I discussed this recently; opening an issue to track it.

### Problem

Many APIs have a limit on the amount of data they can ingest per request. For example, #1187 discusses that the DataDog HTTP API has a 2MB payload limit. A single request is made per flush, and occasionally Fluent Bit can send a chunk which is over 2 MB.

Some APIs have a limit on the number of log messages they can accept per HTTP request. For example, [Amazon CloudWatch](https://github.com/fluent/fluent-bit/issues/1222) has a 10,000 log message limit per PutLogEvents call. Amazon Kinesis Firehose and Amazon Kinesis Data Streams have a much smaller batch limit of 500 events.

Consequently, plugins have to implement logic to split a single chunk into multiple requests (or accept that occasionally large chunks will fail to be sent). This becomes troublesome when a single API request fails in the set. If the plugin issues a retry, the whole chunk will get retried. The fractions of the chunk that got successfully uploaded will thus be sent multiple times.

### Possible Solutions

#### Ideal solution: Output Plugins specify a max chunk size

Ideally, plugins should only have to make a single request per flush. This keeps the logic in the plugin very simple and straightforward. The common task of splitting chunks into right-sized pieces could be placed in the core of Fluent Bit.

Each output plugin could give Fluent Bit a max chunk size.

Implementing this would involve some complexity. Fluent Bit should not allocate additional memory to split chunks into smaller pieces. Instead it can pass a pointer to a fraction of chunk to an output, and track when the entire chunk has successfully been sent.

#### Non-ideal, but easy solution

The most important issue is retries. If each flush had a unique ID associated with it, plugins could internally track whether a flush is a first attempt or a retry, and then track whether the entirety of a chunk had been sent or not.

This is not a good idea, it makes the plugin very complicated; I've included it for the sake of completeness.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.