apache / apache/lucene

LZ4 compression using too much CPU time [LUCENE-6841]

Open
#7,899 3 comments 0 reactions 0 assignees View on GitHub
affects-version:5.3.1 legacy-jira-priority:Major module:core/codecs type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I am using Lucene for search indexing, including storing a large number of small fields, and some larger plain text fields, and searching using both exact matches and analyzed queries.

LZ4 (specifically the decompress method) is using nearly exactly 50% of the application's CPU time.

It seems to me that LZ4 is inappropriate for my use case. I note that I can choose BEST_SPEED or BEST_COMPRESSION.

Would it be palatable to add a NO_COMPRESSION option, or some way to pick and choose which fields get compressed? Perhaps a minimum length of a field could be specified before it's compressed? I'm not sure if that's possible.

If this approach, or similar is palatable, I would be happy to contribute a patch (or to consume and test a patch).

---
Migrated from [LUCENE-6841](https://issues.apache.org/jira/browse/LUCENE-6841) by Karl von Randow, 1 vote
Environment:
```
Linux, Java 8
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing Lucene's LZ4 decompression path and the configuration used for stored fields, then review how BEST_SPEED and BEST_COMPRESSION are selected. Define whether disabling compression, selecting fields, or applying a minimum field length is feasible; done requires an agreed option that addresses the reported CPU cost and tests its behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.