Add block-level compression for ClickHouse sink (ArrowStream)

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Today the only way to compress an ArrowStream payload is a whole-payload HTTP gzip Content-Encoding.

This is effective on size but expensive on CPU, which is important in high-throughput scenarios with ArrowStream.

Ideally we can compress the payload that keeps CPU cost low on the Vector agent and let ClickHouse decompress efficiently on ingest (particularly when egress bandwidth is metered e.g. cross-region / cross-cloud).

Attempted Solutions

Today the only option is a whole-payload HTTP gzip Content-Encoding — good on size, heavy on CPU. Arrow IPC's own block-level (columnar) compression (LZ4/ZSTD) is cheaper but isn't exposed through the sink.

Local benchmark (50k-row batch, ~7.95 MB uncompressed Arrow IPC, single-threaded encode).

On the Clickhouse side, median decode CPU for a 200k INSERT seems to be actually lower compared to the gzip path as well: ZSTD ~14 ms, LZ4 ~18 ms, uncompressed ~18 ms, gzip Content-Encoding ~28 ms

Proposal

Add a compression option to the clickhouse sink's arrow_stream batch encoding, applied as Arrow IPC block compression:

sinks:
  my_clickhouse:
    type: clickhouse
    inputs: [my_source]
    endpoint: http://clickhouse:8123
    database: mydb
    table: mytable
    format: arrow_stream
    batch_encoding:
      codec: arrow_stream
      compression: zstd   # one of: none (default) | lz4_frame | zstd

References

No response

Version

0.57.0

Start in the ClickHouse sink's arrow_stream batch encoding path and inspect how the existing HTTP gzip option and codec configuration are handled. Add support for the proposed none, lz4_frame, and zstd compression values, keeping none as the default, and verify that the resulting ArrowStream payload is accepted by ClickHouse.