ClickHouse / ClickHouse/ClickHouse
A higher order array functions to pack bits.
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
**Use case**
Compression and quantization in vector search databases.
**Describe the solution you'd like**
```
-- fill the bits of UInt64 with the value of a lambda function
-- if the array size is greater than 64, only the first 64 elements are processed
-- if the array size is less than 64, the remaining bits are assumed to be zero
-- the order of bytes is little endian, and the order of bits - as it will be more efficient from the implementation standpoint
arrayPackBitsToUInt64(x -> bit, arr)
-- into a fixed string of specified size in bytes
arrayPackBitsToFixedString(x -> bit, arr, 32)
-- into a string; the length of the string is ceil(array size / 8)
arrayPackBitsToString(x -> bit, arr)
-- for the case when a lambda function returns multiple bits in the form of a number 0..2^n-1,
-- we can pack these groups of bits into the result
-- in this example, we process groups of 4 bits, aka "nibbles".
arrayPackBitGroupsToUInt64(x -> bit, arr, 4)
arrayPackBitGroupsToFixedString(x -> bit, arr, 4, 32)
arrayPackBitGroupsToString(x -> bit, arr, 4)
```
**Describe alternatives you've considered**
This works, but not efficient:
```
unbin(arrayStringConcat(arrayMap(x -> bit, arr)))
```
Contributor guide
Assessment
This issue has not been assessed yet.