ClickHouse / ClickHouse/ClickHouse
Document how to process response blocks in parallel
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
_No response_
### Describe the issue
I get 256 blocks in a response in https://github.com/ClickHouse/ClickHouse/issues/95686. I would like to process them in parallel as processing for each block is not trivial. There's a `LowCardinality(String)` column, so I can't compute the block boundary without processing it fully before continuing to the next block.
There doesn't seem to be a documented way to do this.
So far the best idea I could come up with is:
1. Enable native block compression with `compress=1` (not HTTP compression).
2. Collect frames until the boundary is not at 1MiB (`DBMS_DEFAULT_BUFFER_SIZE`, presumably).
3. Process each collected bunch of compressed frames as possibly multiple blocks.
I'm not sure if this is the best way to do it. It would be nice to have some docs for this.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.