Option for disabling term dictionary compression
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
While working on a customer issue, I noticed that memory allocations for recently added [term dictionary compression](https://github.com/apache/lucene-solr/commit/33a7af9cbfb9f668b4aee433906ee93d55e1e709) is significant. After disabling the compression using patch, I was able to notice some reduction in the memory allocation.
Generally the cost of storage is significantly lower than memory/CPU, but can be useful once the segment/index is being archived. But during live data ingestion when segments merge frequently, the cost of compression/decompression is paid more than once.
Wondering couple of things here:
* Should we expose an option to disable term dictionary compression?
* Does it make sense to initialize the HighCompressionHashTable lazily in TermsWriter? As some code paths (non-compression) don't end up using this.
For context, the customer workload is running on instance having 32G memory with 16G allocated for heap. Attaching the memory allocation profile below:

Contributor guide
Research direction
Begin with TermsWriter and HighCompressionHashTable, then trace the term-dictionary compression paths described in the issue. The work is complete only when the project has a decided way to disable compression and/or initialize the table lazily, with the memory and ingestion trade-offs verified against the reported behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100