ClickHouse / ClickHouse/ClickHouse

Consider Adding Codec to work with Hex Strings

Open
#4,420 3 comments 10 reactions 0 assignees View on GitHub
comp-codecs external feature
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
20h 33m
Merged PRs (30d)
501

Description

(you don't have to strictly follow this form)

**Use case**

Better compression for hex strings, such as "813031B8BBC3B329"

**Describe the solution you'd like**

Hex Strings are used in many applications to represent short binary data in printable form
While ClickHouse has hex() and unhex() functions it is intrusive to the application

**Describe alternatives you've considered**

Some different data type which stores data as binary but displays it in hex would be alternative but I assume it is much more labor intensive while unhex() codec might be easier.

Of course codec would need to be able to reject non-hex data input as invalid in this case.

**Additional context**

Simply storing hex strings as strings compared to converting them to binary results in 10-15% performance overhead, which is not significant. However column also takes some 80% more on disk using LZ4 compression which to my surprise is unable to compress such data well.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.