ClickHouse / ClickHouse/ClickHouse

ArrowStream emits thousands of tiny record batches, adding substantial decoding overhead

Open
#119,815 3 comments 0 reactions 0 assignees View on GitHub
comp-formats external unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

_No response_

### Describe the unexpected behaviour

Sparse SELECTs can produce thousands of tiny Arrow record batches, making the result much more expensive to transmit and decode than its row count suggests.

One query returned 16,521 rows across 32 columns in **4,148 record batches**: median 2 rows, maximum 40. To isolate the batching cost, we reencoded the same ordered result using the same Arrow writer and LZ4 settings:

| Layout of the same result | IPC size | Processing in libarrow C++ |
|---|---:|---:|
| 4,148 record batches | 24.7 MB | 118–121 ms |
| 1 record batch | 4.9 MB | 3.02–3.05 ms |

Processing here means IPC decoding and table construction, not including query execution. Both layouts used the same decoder and settings, after some warmup.

### Which ClickHouse versions are affected?

Reproduced on **26.7.2.59**. Older ClickHouse binaries have not been tested, but the historical libarrow-based writer also lacked cross-chunk coalescing, so this is likely not a new problem.

### How to reproduce

Send this query over HTTP and save the ArrowStream response:

```sql
SELECT number
FROM numbers(268435456)
WHERE number % 65536 = 0
FORMAT ArrowStream
```

By contrast, this generates exactly the same ordered values directly:

```sql
SELECT n.number * 65536 AS number
FROM numbers(4096) AS n
FORMAT ArrowStream
```

Observed responses:

| Query | Result rows | Record batches | IPC bytes |
|---|---:|---:|---:|
| Sparse filter | 4,096 | 4,096, one row each | 721,040 |
| Direct generator | 4,096 | 1 | 16,728 |

Relevant settings were:

```text
max_block_size = 65409
preferred_block_size_bytes = 1000000
output_format_arrow_compression_method = lz4_frame
```

### Expected behavior

An output-side way to coalesce small Arrow batches independently of query-processing block sizes, without changing scan or aggregation settings or adding an otherwise unnecessary sort.

This need not mean buffering the entire result. Configurable coalescing could retain incremental output while trading some first-batch latency and memory for batching efficiency.

### Error message and/or stacktrace

_No response_

### Related issues and pull requests

Related: https://github.com/ClickHouse/ClickHouse/issues/95686

Related: https://github.com/ClickHouse/ClickHouse/pull/107897

### Additional context

[`ArrowIPCBlockOutputFormat::consume`](https://github.com/ClickHouse/ClickHouse/blob/v26.7.2.59-stable/src/Processors/Formats/Impl/ArrowIPC/ArrowIPCBlockOutputFormat.cpp) writes one record batch per incoming chunk. Small chunks left by filtering therefore become small Arrow batches. This produces huge Arrow IPC overhead, and consumers such as libarrow C++ repeatedly pay for metadata handling, buffer/array allocation, and decompression setup.

One possible approach would be to stage small chunks in the native Arrow formatter until a row or byte target is reached. [`ParquetBlockOutputFormat`](https://github.com/ClickHouse/ClickHouse/blob/v26.7.2.59-stable/src/Processors/Formats/Impl/ParquetBlockOutputFormat.cpp) does this, for example. Another option is to add a squashing transform before producing the final output.

The related block-size issue discusses the upstream execution tradeoffs. The proposed Arrow `row_group_size` setting in #107897 concerns splitting larger blocks, rather than combining smaller ones.

Contributor guide

Open the contributing guide

Research direction

Start in src/Processors/Formats/Impl/ArrowIPC/ArrowIPCBlockOutputFormat.cpp, especially ArrowIPCBlockOutputFormat::consume, and compare its per-chunk writing with src/Processors/Formats/Impl/ParquetBlockOutputFormat.cpp. Reproduce the sparse numbers query over HTTP with FORMAT ArrowStream and the listed settings. Done means small chunks are coalesced for Arrow output without changing query-processing block sizes, while output remains incremental.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.