fluent / fluent/fluent-bit

out_http: add an option to batch multiple chunks into a single request

Open
#12,253 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

**Is your feature request related to a problem? Please describe.**
I ship haproxy access logs to ClickHouse using the http output (`format json_stream`, the URI does `INSERT ... FORMAT JSONEachRow`). I set `flush 20` expecting one insert every 20 seconds, but I actually get around 5 requests on every flush, each about 2MB.

After reading the source I understand why: out_http sends one HTTP request per chunk, and input chunks are capped at ~2MB (`FLB_INPUT_CHUNK_FS_MAX_SIZE`). So the flush interval only controls *when* requests are sent, not *how many*. Any volume above 2MB per interval means multiple requests, and there is no option to change this. For ClickHouse many small inserts are a known problem (too many parts), but this applies to any http endpoint that prefers fewer, larger requests. The closest thing I found in the tracker is #551, which started with the same wish (one POST per interval, or earlier at a size limit) but was answered as a question about tag grouping and never became a feature.

**Describe the solution you'd like**
An opt-in batching mode in out_http, similar to what out_s3 already does with its staging buffer (`total_file_size` / `upload_timeout`), just simpler:

```
[OUTPUT]
name http
match *
host my-server
port 8123
format json_stream
batch_size 32M
batch_timeout 20s
```

- accumulate incoming chunks (per tag) and send them as one request when `batch_size` bytes are buffered or the oldest data is `batch_timeout` old
- if `batch_timeout` is not set, default it to the service `flush` interval, so `flush 20` really means one request per 20 seconds
- with both options unset the behavior stays exactly as today (one request per chunk)

**Describe alternatives you've considered**
- ClickHouse `async_insert` works for my case, but it is server side and doesn't help with other http endpoints
- raising the 2MB chunk cap is not possible without recompiling, it is a compile-time constant

Contributor guide

Open the contributing guide

Research direction

Begin with the out_http source and compare out_s3's staging-buffer behavior, including how flush timing and FLB_INPUT_CHUNK_FS_MAX_SIZE affect requests. Done means opt-in batch_size and batch_timeout behavior matches the requested size or age triggers, while leaving the current one-request-per-chunk behavior unchanged when both options are unset.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
api, backend, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.