apache / apache/pinot

Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index

Open
#6,804 9 comments 3 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

When the **forward index is not dictionary encoded**, we have 2 choices:

- store the data as is (RAW)
- store the data **snappy** compressed - using snappy compression codec library

In addition to snappy, we should add support for other compression codecs subject to their availability in Java libraries.

Currently by default we use Snappy compression. However, this didn't really give good compression ratio for free-text data. LZO is known to provide better compression ratio and speed for larger char/varchar data.

So, we should explore other options

Firstly, we should start with simple test case to compress and uncompress direct byte buffer and do some functional and performance tests.

see ZSTD library in Java - https://github.com/luben/zstd-jni

Any new ideas/suggestions?

Contributor guide

Open the contributing guide

Research direction

Start by reading the raw forward-index handling and the existing Snappy compression path, then create the proposed direct-byte-buffer compression and decompression test. Investigate Java libraries for LZO, LZ4, ZSTD, DEFLATE, and GZIP, and compare functional behavior and performance. Done means the supported codec options and test results are documented clearly enough to guide implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.