out_http: add an option to batch multiple chunks into a single request
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
**Is your feature request related to a problem? Please describe.**
I ship haproxy access logs to ClickHouse using the http output (`format json_stream`, the URI does `INSERT ... FORMAT JSONEachRow`). I set `flush 20` expecting one insert every 20 seconds, but I actually get around 5 requests on every flush, each about 2MB.
After reading the source I understand why: out_http sends one HTTP request per chunk, and input chunks are capped at ~2MB (`FLB_INPUT_CHUNK_FS_MAX_SIZE`). So the flush interval only controls *when* requests are sent, not *how many*. Any volume above 2MB per interval means multiple requests, and there is no option to change this. For ClickHouse many small inserts are a known problem (too many parts), but this applies to any http endpoint that prefers fewer, larger requests. The closest thing I found in the tracker is #551, which started with the same wish (one POST per interval, or earlier at a size limit) but was answered as a question about tag grouping and never became a feature.
**Describe the solution you'd like**
An opt-in batching mode in out_http, similar to what out_s3 already does with its staging buffer (`total_file_size` / `upload_timeout`), just simpler:
```
[OUTPUT]
name http
match *
host my-server
port 8123
format json_stream
batch_size 32M
batch_timeout 20s
```
- accumulate incoming chunks (per tag) and send them as one request when `batch_size` bytes are buffered or the oldest data is `batch_timeout` old
- if `batch_timeout` is not set, default it to the service `flush` interval, so `flush 20` really means one request per 20 seconds
- with both options unset the behavior stays exactly as today (one request per chunk)
**Describe alternatives you've considered**
- ClickHouse `async_insert` works for my case, but it is server side and doesn't help with other http endpoints
- raising the 2MB chunk cap is not possible without recompiling, it is a compile-time constant
Contributor guide
Research direction
Begin with the out_http source and compare out_s3's staging-buffer behavior, including how flush timing and FLB_INPUT_CHUNK_FS_MAX_SIZE affect requests. Done means opt-in batch_size and batch_timeout behavior matches the requested size or age triggers, while leaving the current one-request-per-chunk behavior unchanged when both options are unset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- api, backend, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100