apache / apache/lucene

Add compression feature for DocValues format in new Codec

Open
#12,137 1 comment 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

We use ES as an OLAP engine in advertising scenarios, an advertiser will query the data of his own. We usually make advertiser_id as a routing and index sorting key so the read density is very high in docvalues. We leverage lucene's posting index structure to speedup the query and the performance meet our expectation.

The most complained part of ES/lucene is that the disk usage is much bigger than clickhouse/doris, and in our case, lucene storage can be 3-4x times bigger.

The reason why clickhouse/doris performs better in space is that they both compress data in blocks and uncompress the needed blocks on read. Since the read density is high, the performance is still acceptable.

We also implement the zstd/lz4 compression for lucene docvalues, below is the storage cost improvement:

 name | total size| docvalue size | docvalue compression ratio
-- | -- | -- | --
no compression | 485.8g | 394.6g   | 100%
lz4 | 272g | 255.1g | 64.65% | 64.65%
zstd | 246.5g | 229.5g | 58.16%

All the docvalues is numeric and we compress the data in block of 4096 values.

We also run a high QPS(4000) load test from our online query set, the pct50 and pct99 both decreases by 20% to 30%. We are shocked by this improvement and we guess the read density is the key to the result.

But the disadvantage of compression is that it hurts the performance of random read a lot because a full block of data must be read ahead even just only 1 byte is needed.

I suggest that we create a new codec for compression and the purpose of this codec is to reduce the storage usage and provide adequate and balanced read performance is OLAP cases.

Contributor guide

Open the contributing guide

Research direction

The proposal concerns Lucene DocValues and a new compression-focused codec, but names no files, tests, or entry points. Start by locating the existing DocValues codec implementations and their tests, then define the codec's supported formats and validation for storage usage and read performance before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.