apache / apache/lucene

Add NO_COMPRESSION option to compression Mode

Open
#12,424 2 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Our system ran on Lucene 3x for years without issues. We recently upgraded to 9.4 and have had nothing but trouble with performance. Our system reads each returned document every search to provide a result clustering mechanism (similar products merge to one result). indexSearch.doc seems to be so much slower than before and it seems to be caused by the compression stuff or some problem with internal resource contention once we get over a few hundred concurrent searches. Every one of our threads gets stuck in org.apache.lucene.util.compress.LZ4.decompress if we are not careful.

In my studies it seems you may decompress in blocks, and you can imagine how painful this is when our most common searches return a lot of results and need filtering. We already have the fast decompression setting, but as many people online have discussed, I would like a global NO_COMPRESSION setting. I am glad to throw RAM at the problem. It seems some have been able to solve it by implementing their own FilterCodec, but that seems so wacky. Why not let us disable with a setting so our use case can be supported?

I probably could go contribute this fix, but it seems like the decision is more philosophical than technical. I don't want to put in the effort if it is just going to be rejected. I am hoping that this request can either convince your team to make this hopefully simple change, or give me the encouragement to attempt my own submission.

Thanks!

Contributor guide

Open the contributing guide

Research direction

Start at indexSearch.doc and trace the path into org.apache.lucene.util.compress.LZ4.decompress, comparing it with the existing fast decompression setting and the custom FilterCodec approach mentioned in the issue. Done means agreeing on and implementing a global NO_COMPRESSION option that supports this workload without requiring a custom codec.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.