ClickHouse / ClickHouse/ClickHouse

Experiment: compression codecs with separate dictionaries.

Open
#110,611 4 comments 2 reactions 0 assignees View on GitHub
comp-codecs feature research
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

ClickHouse

### Use case

Improving compression ratio, and, sometimes, speed.

### Describe the solution you'd like

Most compression algorithms accumulate some context during their operation and allow for preparing this context in advance and reusing it multiple times for multiple blocks. It is named compression with a dictionary, or sometimes, pretrained compression. For example, for LZ77, it is the contents of the sliding window plus the current state of the hash table for match search.

We compress data in fully independent blocks in ClickHouse, which allows seeking inside the compressed file. Different compressed blocks do not reuse any information from each other.

A natural idea is to reuse dictionaries between different blocks to improve the compression ratio. But there are implementation details - how to collect, store, and load these dictionaries.

We already have the required building blocks for these implementation details. For example, LowCardinality, Dynamic, and JSON columns have a separate stream with the data, which has to be loaded in advance before loading other data streams with values. We can use a similar principle for compression with dictionaries.

Let's say a compression CODEC will accept parameters to enable dictionaries. If a column is written with this compression codec, we will write additional files for each compressed binary file. For example, when the main file is Column.bin, the additional file will be Column.bin.dict. The dictionary is created from the first compressed block (so it is "trained" on a single block only) and written (or sent over the network) before the main file. When reading data, we locate the dictionary and read it before the main file.

The `ICompressionCodec` interface will be changed to accept an additional ReadBuffer/WriteBuffer to read or write the dictionary. The dictionary will be read/written once and retained in memory during the processing of the whole data stream, containing multiple compressed blocks, reusing the same dictionary.

### Describe alternatives you've considered

The benefits will be marginal, so the main goal is to research "what if".

### Additional context

Custom codecs may use more interesting and specialized methods for constructing their dictionaries.

We can also consider cross-column compression (co-compressed columns) - when the state after compressing a block for one column is reused for compressing another column. This improves compression of dependent columns, like domain and URL, or date and datetime.

Contributor guide

Open the contributing guide

Research direction

Start by reading the ICompressionCodec interface and the existing separate-stream handling for LowCardinality, Dynamic, and JSON columns. Compare the proposed Column.bin and Column.bin.dict storage and loading flow, including retention across compressed blocks. Done means establishing whether dictionary reuse improves compression ratio or speed and identifying the required design boundaries.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.