ClickHouse / ClickHouse/ClickHouse

Document how to process response blocks in parallel

Open
#96,704 0 comments 0 reactions 0 assignees View on GitHub
comp-documentation external
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

_No response_

### Describe the issue

I get 256 blocks in a response in https://github.com/ClickHouse/ClickHouse/issues/95686. I would like to process them in parallel as processing for each block is not trivial. There's a `LowCardinality(String)` column, so I can't compute the block boundary without processing it fully before continuing to the next block.

There doesn't seem to be a documented way to do this.

So far the best idea I could come up with is:

1. Enable native block compression with `compress=1` (not HTTP compression).
2. Collect frames until the boundary is not at 1MiB (`DBMS_DEFAULT_BUFFER_SIZE`, presumably).
3. Process each collected bunch of compressed frames as possibly multiple blocks.

I'm not sure if this is the best way to do it. It would be nice to have some docs for this.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.