Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
When the **forward index is not dictionary encoded**, we have 2 choices:
- store the data as is (RAW)
- store the data **snappy** compressed - using snappy compression codec library
In addition to snappy, we should add support for other compression codecs subject to their availability in Java libraries.
Currently by default we use Snappy compression. However, this didn't really give good compression ratio for free-text data. LZO is known to provide better compression ratio and speed for larger char/varchar data.
So, we should explore other options
Firstly, we should start with simple test case to compress and uncompress direct byte buffer and do some functional and performance tests.
see ZSTD library in Java - https://github.com/luben/zstd-jni
Any new ideas/suggestions?
Contributor guide
Research direction
Start by reading the raw forward-index handling and the existing Snappy compression path, then create the proposed direct-byte-buffer compression and decompression test. Investigate Java libraries for LZO, LZ4, ZSTD, DEFLATE, and GZIP, and compare functional behavior and performance. Done means the supported codec options and test results are documented clearly enough to guide implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100