Allow output plugins to configure a max chunk size
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
@edsiper and I discussed this recently; opening an issue to track it.
### Problem
Many APIs have a limit on the amount of data they can ingest per request. For example, #1187 discusses that the DataDog HTTP API has a 2MB payload limit. A single request is made per flush, and occasionally Fluent Bit can send a chunk which is over 2 MB.
Some APIs have a limit on the number of log messages they can accept per HTTP request. For example, [Amazon CloudWatch](https://github.com/fluent/fluent-bit/issues/1222) has a 10,000 log message limit per PutLogEvents call. Amazon Kinesis Firehose and Amazon Kinesis Data Streams have a much smaller batch limit of 500 events.
Consequently, plugins have to implement logic to split a single chunk into multiple requests (or accept that occasionally large chunks will fail to be sent). This becomes troublesome when a single API request fails in the set. If the plugin issues a retry, the whole chunk will get retried. The fractions of the chunk that got successfully uploaded will thus be sent multiple times.
### Possible Solutions
#### Ideal solution: Output Plugins specify a max chunk size
Ideally, plugins should only have to make a single request per flush. This keeps the logic in the plugin very simple and straightforward. The common task of splitting chunks into right-sized pieces could be placed in the core of Fluent Bit.
Each output plugin could give Fluent Bit a max chunk size.
Implementing this would involve some complexity. Fluent Bit should not allocate additional memory to split chunks into smaller pieces. Instead it can pass a pointer to a fraction of chunk to an output, and track when the entire chunk has successfully been sent.
#### Non-ideal, but easy solution
The most important issue is retries. If each flush had a unique ID associated with it, plugins could internally track whether a flush is a first attempt or a retry, and then track whether the entirety of a chunk had been sent or not.
This is not a good idea, it makes the plugin very complicated; I've included it for the sake of completeness.
Contributor guide
Assessment
This issue has not been assessed yet.