Add NO_COMPRESSION option to compression Mode
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Our system ran on Lucene 3x for years without issues. We recently upgraded to 9.4 and have had nothing but trouble with performance. Our system reads each returned document every search to provide a result clustering mechanism (similar products merge to one result). indexSearch.doc seems to be so much slower than before and it seems to be caused by the compression stuff or some problem with internal resource contention once we get over a few hundred concurrent searches. Every one of our threads gets stuck in org.apache.lucene.util.compress.LZ4.decompress if we are not careful.
In my studies it seems you may decompress in blocks, and you can imagine how painful this is when our most common searches return a lot of results and need filtering. We already have the fast decompression setting, but as many people online have discussed, I would like a global NO_COMPRESSION setting. I am glad to throw RAM at the problem. It seems some have been able to solve it by implementing their own FilterCodec, but that seems so wacky. Why not let us disable with a setting so our use case can be supported?
I probably could go contribute this fix, but it seems like the decision is more philosophical than technical. I don't want to put in the effort if it is just going to be rejected. I am hoping that this request can either convince your team to make this hopefully simple change, or give me the encouragement to attempt my own submission.
Thanks!
Contributor guide
Research direction
Start at indexSearch.doc and trace the path into org.apache.lucene.util.compress.LZ4.decompress, comparing it with the existing fast decompression setting and the custom FilterCodec approach mentioned in the issue. Done means agreeing on and implementing a global NO_COMPRESSION option that supports this workload without requiring a custom codec.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100