apache / apache/lucene

Option for disabling term dictionary compression

Open
#12,317 6 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

While working on a customer issue, I noticed that memory allocations for recently added [term dictionary compression](https://github.com/apache/lucene-solr/commit/33a7af9cbfb9f668b4aee433906ee93d55e1e709) is significant. After disabling the compression using patch, I was able to notice some reduction in the memory allocation.

Generally the cost of storage is significantly lower than memory/CPU, but can be useful once the segment/index is being archived. But during live data ingestion when segments merge frequently, the cost of compression/decompression is paid more than once.

Wondering couple of things here:

* Should we expose an option to disable term dictionary compression?
* Does it make sense to initialize the HighCompressionHashTable lazily in TermsWriter? As some code paths (non-compression) don't end up using this.

For context, the customer workload is running on instance having 32G memory with 16G allocated for heap. Attaching the memory allocation profile below:

![Screenshot 2023-05-19 at 5 27 53 PM](https://github.com/apache/lucene/assets/8193480/bfa964b4-76d4-4903-89a7-7164f993ea3e)

Contributor guide

Open the contributing guide

Research direction

Begin with TermsWriter and HighCompressionHashTable, then trace the term-dictionary compression paths described in the issue. The work is complete only when the project has a decided way to disable compression and/or initialize the table lazily, with the memory and ingestion trade-offs verified against the reported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.