ClickHouse / ClickHouse/ClickHouse

A higher order array functions to pack bits.

Open
#48,830 5 comments 0 reactions 1 assignee Claimed by @Eridanus117 View on GitHub
comp-regular-function feature warmup task
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

**Use case**

Compression and quantization in vector search databases.

**Describe the solution you'd like**

```
-- fill the bits of UInt64 with the value of a lambda function
-- if the array size is greater than 64, only the first 64 elements are processed
-- if the array size is less than 64, the remaining bits are assumed to be zero
-- the order of bytes is little endian, and the order of bits - as it will be more efficient from the implementation standpoint
arrayPackBitsToUInt64(x -> bit, arr)

-- into a fixed string of specified size in bytes
arrayPackBitsToFixedString(x -> bit, arr, 32)

-- into a string; the length of the string is ceil(array size / 8)
arrayPackBitsToString(x -> bit, arr)

-- for the case when a lambda function returns multiple bits in the form of a number 0..2^n-1,
-- we can pack these groups of bits into the result
-- in this example, we process groups of 4 bits, aka "nibbles".
arrayPackBitGroupsToUInt64(x -> bit, arr, 4)
arrayPackBitGroupsToFixedString(x -> bit, arr, 4, 32)
arrayPackBitGroupsToString(x -> bit, arr, 4)
```

**Describe alternatives you've considered**

This works, but not efficient:
```
unbin(arrayStringConcat(arrayMap(x -> bit, arr)))
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.