complex Query cause luene outMemory [LUCENE-5917]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
RangeQuery, prefixQuery and WildcardQuery use FixedBitSet when TERM_COUNT >= 350 or DOC_COUNT_PERCENT >=0.1.
It use a lots of memory when maxDoc very large.
MultiTermQueryWrapperFilter<Q extends MultiTermQuery> extends Filter
a little threads run with query "a\* OR b\* OR c\*.....OR z\*“ will cause luene outMemory, but there is no ways to prevent it.
another thing, some complex query also use a lot of memory too..
I think query implements Accountable(#ramSizeInBytes, #ramBytesUsed), users can throw a exception better than OutOfMemory.
---
Migrated from [LUCENE-5917](https://issues.apache.org/jira/browse/LUCENE-5917) by Littlestar
Environment:
```
256G RAM + 1.5T SSD + 1.2T DATA(10*10000*10000 records).
java heap=128G, G1 GC
```
Contributor guide
Research direction
Start with the reported RangeQuery, PrefixQuery, WildcardQuery, and MultiTermQueryWrapperFilter classes, then review the linked LUCENE-5917 report and its memory conditions. Reproduce the complex wildcard-query case if possible and determine how memory usage should be accounted for; done requires an agreed way to prevent or report exhaustion without relying on an OutOfMemoryError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100