A little optimization about LZ4
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Since lucene8.7, we use preset dictionaries with LZ4 for stored fields https://issues.apache.org/jira/browse/LUCENE-9486, as the result:
- The compression ratio witnesses respective improvements.
- It makes fetch times a bit slower.
When reading every doc, we will read dictionaries first, and then the doc, which will use more IO. In our product, we upgrade lucene from 8.6 to 9.x, and observed that IOPs and IO throughput usage had doubled. After inspection, we ascertained that it was the consequence of the introduction of preset dictionaries.
Additionally, I conducted test with `pmc` dataset, the compression ratio has been improved by 11.7% with preset dict about `fdt`.
| Method | Index size(MB) |
| ------------- | ------------- |
| LZ4 with no preset dict (BEST_SPEED 16k, 128 doc) | 2055 |
| LZ4 with preset dict (BEST_SPEED 80k, 1024 doc) | 1813 |
### Describe the solution you'd like
1. Given these findings, should we consider reusing the LZ4 with no preset dict algorithm as a viable alternative? User can select this compression algorithm through parameters to reduce disk io.
2. Compressing: we will reset `Dictionary` for each sub block, for faster compressing, can we consider using space in exchange for time: Reserving a backup dictionary, cloning for each sub block. we can potentially reduce the time spent on resetting 10 dictionaries.
3. Decompressing: we reading every doc, firstly we must read the total `preset dictionaries`, then read the doc content. If the doc is small compared with dictionaries, which leads to significant wastage of both I/O resources and time. If we can reserve the `preset dictionaries` and `decompressed sub chunk content` for the last chunk, If the chunk of next doc is same, we can reuse the `preset dictionaries` and `decompressed sub chunk content` directly, instead of regenerating the same `preset dictionaries`.
Contributor guide
Research direction
Start by tracing Lucene's stored-field LZ4 compression and decompression paths, including preset-dictionary handling, and reproduce the reported pmc/fdt compression and I/O observations. Done would require a decided scope among the proposed alternatives, with tests or benchmarks showing the effect on compression ratio, fetch time, and I/O.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100