Drop unnecessary round-trip in ClickHouse Arrow encoding

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

The clickhouse sink's arrow_stream encoding is used for high-volume ingestion, so its encode path runs on every flushed batch. Cutting per-event CPU there directly lowers the cores needed to sustain a given throughput.

Attempted Solutions

Today the encoder makes two passes over every event:

1. Convert each event into a serde_json::Value tree.
2. Feed those trees to Arrow's JSON decoder to build the columnar batch.

Step 1 isn't needed, since Arrow's decoder accepts anything that implements Serialize, and LogEvent already implements it, meaning that the intermediate JSON tree is unnecessary.

Proposal

Serialize events straight into the Arrow decoder, skipping the serde_json::Value tree. This drops one allocation and one serialization pass per event.

• No config, schema, or API changes.
• Behavior-preserving: all type coercion still runs through the same, unchanged Arrow decoder.

On a local bench, ~10 MB batch of 8,000 wide/nested events:

Wider and more deeply nested events see the largest improvement, since they are the most expensive to convert into intermediate JSON.

This should also have the benefit of reducing some LOC / complexity that was added via the initial implementation in #24373

References

• ClickHouse sink ArrowStream support: #24074 (request)
• Partially addresses #25734, which removes the first step where every event is first materialized as an intermediate JSON value.

Version

0.57.0

Start at the ClickHouse sink's arrow_stream encoding path and inspect how LogEvent values reach Arrow's decoder. Replace the intermediate serde_json::Value step while preserving the existing decoder and type coercion behavior; compare the local benchmark for an approximately 10 MB batch of 8,000 wide or nested events to confirm the round-trip is removed and performance improves.