ClickHouse / ClickHouse/ClickHouse
Experiment: compression codecs with separate dictionaries.
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
ClickHouse
### Use case
Improving compression ratio, and, sometimes, speed.
### Describe the solution you'd like
Most compression algorithms accumulate some context during their operation and allow for preparing this context in advance and reusing it multiple times for multiple blocks. It is named compression with a dictionary, or sometimes, pretrained compression. For example, for LZ77, it is the contents of the sliding window plus the current state of the hash table for match search.
We compress data in fully independent blocks in ClickHouse, which allows seeking inside the compressed file. Different compressed blocks do not reuse any information from each other.
A natural idea is to reuse dictionaries between different blocks to improve the compression ratio. But there are implementation details - how to collect, store, and load these dictionaries.
We already have the required building blocks for these implementation details. For example, LowCardinality, Dynamic, and JSON columns have a separate stream with the data, which has to be loaded in advance before loading other data streams with values. We can use a similar principle for compression with dictionaries.
Let's say a compression CODEC will accept parameters to enable dictionaries. If a column is written with this compression codec, we will write additional files for each compressed binary file. For example, when the main file is Column.bin, the additional file will be Column.bin.dict. The dictionary is created from the first compressed block (so it is "trained" on a single block only) and written (or sent over the network) before the main file. When reading data, we locate the dictionary and read it before the main file.
The `ICompressionCodec` interface will be changed to accept an additional ReadBuffer/WriteBuffer to read or write the dictionary. The dictionary will be read/written once and retained in memory during the processing of the whole data stream, containing multiple compressed blocks, reusing the same dictionary.
### Describe alternatives you've considered
The benefits will be marginal, so the main goal is to research "what if".
### Additional context
Custom codecs may use more interesting and specialized methods for constructing their dictionaries.
We can also consider cross-column compression (co-compressed columns) - when the state after compressing a block for one column is reused for compressing another column. This improves compression of dependent columns, like domain and URL, or date and datetime.
Contributor guide
Research direction
Start by reading the ICompressionCodec interface and the existing separate-stream handling for LowCardinality, Dynamic, and JSON columns. Compare the proposed Column.bin and Column.bin.dict storage and loading flow, including retention across compressed blocks. Done means establishing whether dictionary reuse improves compression ratio or speed and identifying the required design boundaries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100