ClickHouse / ClickHouse/ClickHouse
ArrowStream emits thousands of tiny record batches, adding substantial decoding overhead
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
_No response_
### Describe the unexpected behaviour
Sparse SELECTs can produce thousands of tiny Arrow record batches, making the result much more expensive to transmit and decode than its row count suggests.
One query returned 16,521 rows across 32 columns in **4,148 record batches**: median 2 rows, maximum 40. To isolate the batching cost, we reencoded the same ordered result using the same Arrow writer and LZ4 settings:
| Layout of the same result | IPC size | Processing in libarrow C++ |
|---|---:|---:|
| 4,148 record batches | 24.7 MB | 118–121 ms |
| 1 record batch | 4.9 MB | 3.02–3.05 ms |
Processing here means IPC decoding and table construction, not including query execution. Both layouts used the same decoder and settings, after some warmup.
### Which ClickHouse versions are affected?
Reproduced on **26.7.2.59**. Older ClickHouse binaries have not been tested, but the historical libarrow-based writer also lacked cross-chunk coalescing, so this is likely not a new problem.
### How to reproduce
Send this query over HTTP and save the ArrowStream response:
```sql
SELECT number
FROM numbers(268435456)
WHERE number % 65536 = 0
FORMAT ArrowStream
```
By contrast, this generates exactly the same ordered values directly:
```sql
SELECT n.number * 65536 AS number
FROM numbers(4096) AS n
FORMAT ArrowStream
```
Observed responses:
| Query | Result rows | Record batches | IPC bytes |
|---|---:|---:|---:|
| Sparse filter | 4,096 | 4,096, one row each | 721,040 |
| Direct generator | 4,096 | 1 | 16,728 |
Relevant settings were:
```text
max_block_size = 65409
preferred_block_size_bytes = 1000000
output_format_arrow_compression_method = lz4_frame
```
### Expected behavior
An output-side way to coalesce small Arrow batches independently of query-processing block sizes, without changing scan or aggregation settings or adding an otherwise unnecessary sort.
This need not mean buffering the entire result. Configurable coalescing could retain incremental output while trading some first-batch latency and memory for batching efficiency.
### Error message and/or stacktrace
_No response_
### Related issues and pull requests
Related: https://github.com/ClickHouse/ClickHouse/issues/95686
Related: https://github.com/ClickHouse/ClickHouse/pull/107897
### Additional context
[`ArrowIPCBlockOutputFormat::consume`](https://github.com/ClickHouse/ClickHouse/blob/v26.7.2.59-stable/src/Processors/Formats/Impl/ArrowIPC/ArrowIPCBlockOutputFormat.cpp) writes one record batch per incoming chunk. Small chunks left by filtering therefore become small Arrow batches. This produces huge Arrow IPC overhead, and consumers such as libarrow C++ repeatedly pay for metadata handling, buffer/array allocation, and decompression setup.
One possible approach would be to stage small chunks in the native Arrow formatter until a row or byte target is reached. [`ParquetBlockOutputFormat`](https://github.com/ClickHouse/ClickHouse/blob/v26.7.2.59-stable/src/Processors/Formats/Impl/ParquetBlockOutputFormat.cpp) does this, for example. Another option is to add a squashing transform before producing the final output.
The related block-size issue discusses the upstream execution tradeoffs. The proposed Arrow `row_group_size` setting in #107897 concerns splitting larger blocks, rather than combining smaller ones.
Contributor guide
Research direction
Start in src/Processors/Formats/Impl/ArrowIPC/ArrowIPCBlockOutputFormat.cpp, especially ArrowIPCBlockOutputFormat::consume, and compare its per-chunk writing with src/Processors/Formats/Impl/ParquetBlockOutputFormat.cpp. Reproduce the sparse numbers query over HTTP with FORMAT ArrowStream and the listed settings. Done means small chunks are coalesced for Arrow output without changing query-processing block sizes, while output remains incremental.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100