Add compression feature for DocValues format in new Codec
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
We use ES as an OLAP engine in advertising scenarios, an advertiser will query the data of his own. We usually make advertiser_id as a routing and index sorting key so the read density is very high in docvalues. We leverage lucene's posting index structure to speedup the query and the performance meet our expectation.
The most complained part of ES/lucene is that the disk usage is much bigger than clickhouse/doris, and in our case, lucene storage can be 3-4x times bigger.
The reason why clickhouse/doris performs better in space is that they both compress data in blocks and uncompress the needed blocks on read. Since the read density is high, the performance is still acceptable.
We also implement the zstd/lz4 compression for lucene docvalues, below is the storage cost improvement:
name | total size| docvalue size | docvalue compression ratio
-- | -- | -- | --
no compression | 485.8g | 394.6g | 100%
lz4 | 272g | 255.1g | 64.65% | 64.65%
zstd | 246.5g | 229.5g | 58.16%
All the docvalues is numeric and we compress the data in block of 4096 values.
We also run a high QPS(4000) load test from our online query set, the pct50 and pct99 both decreases by 20% to 30%. We are shocked by this improvement and we guess the read density is the key to the result.
But the disadvantage of compression is that it hurts the performance of random read a lot because a full block of data must be read ahead even just only 1 byte is needed.
I suggest that we create a new codec for compression and the purpose of this codec is to reduce the storage usage and provide adequate and balanced read performance is OLAP cases.
Contributor guide
Research direction
The proposal concerns Lucene DocValues and a new compression-focused codec, but names no files, tests, or entry points. Start by locating the existing DocValues codec implementations and their tests, then define the codec's supported formats and validation for storage usage and read performance before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100