LZ4 compression using too much CPU time [LUCENE-6841]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I am using Lucene for search indexing, including storing a large number of small fields, and some larger plain text fields, and searching using both exact matches and analyzed queries.
LZ4 (specifically the decompress method) is using nearly exactly 50% of the application's CPU time.
It seems to me that LZ4 is inappropriate for my use case. I note that I can choose BEST_SPEED or BEST_COMPRESSION.
Would it be palatable to add a NO_COMPRESSION option, or some way to pick and choose which fields get compressed? Perhaps a minimum length of a field could be specified before it's compressed? I'm not sure if that's possible.
If this approach, or similar is palatable, I would be happy to contribute a patch (or to consume and test a patch).
---
Migrated from [LUCENE-6841](https://issues.apache.org/jira/browse/LUCENE-6841) by Karl von Randow, 1 vote
Environment:
```
Linux, Java 8
```
Contributor guide
Research direction
Start by tracing Lucene's LZ4 decompression path and the configuration used for stored fields, then review how BEST_SPEED and BEST_COMPRESSION are selected. Define whether disabling compression, selecting fields, or applying a minimum field length is feasible; done requires an agreed option that addresses the reported CPU cost and tests its behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100