apache / apache/lucene

complex Query cause luene outMemory [LUCENE-5917]

Open
#6,979 2 comments 0 reactions 0 assignees View on GitHub
affects-version:4.9 legacy-jira-priority:Minor module:core/search type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

RangeQuery, prefixQuery and WildcardQuery use FixedBitSet when TERM_COUNT >= 350 or DOC_COUNT_PERCENT >=0.1.
It use a lots of memory when maxDoc very large.

MultiTermQueryWrapperFilter<Q extends MultiTermQuery> extends Filter

a little threads run with query "a\* OR b\* OR c\*.....OR z\*“ will cause luene outMemory, but there is no ways to prevent it.

another thing, some complex query also use a lot of memory too..

I think query implements Accountable(#ramSizeInBytes, #ramBytesUsed), users can throw a exception better than OutOfMemory.

---
Migrated from [LUCENE-5917](https://issues.apache.org/jira/browse/LUCENE-5917) by Littlestar
Environment:
```
256G RAM + 1.5T SSD + 1.2T DATA(10*10000*10000 records).
java heap=128G, G1 GC

```

Contributor guide

Open the contributing guide

Research direction

Start with the reported RangeQuery, PrefixQuery, WildcardQuery, and MultiTermQueryWrapperFilter classes, then review the linked LUCENE-5917 report and its memory conditions. Reproduce the complex wildcard-query case if possible and determine how memory usage should be accounted for; done requires an agreed way to prevent or report exhaustion without relying on an OutOfMemoryError.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.