apache / apache/lucene

Provide the LeafSlice to CollectorManager.newCollector to save memory on small index slices [LUCENE-8542]

Open
#9,588 17 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I have an index consisting of 44 million documents spread across 60 segments. When I run a query against this index with a huge number of results requested (e.g. 5 million), this query uses more than 5 GB of heap if the IndexSearch was configured to use an ExecutorService.

(I know this kind of query is fairly unusual and it would be better to use paging and searchAfter, but our architecture does not allow this at the moment.)

The reason for the huge memory requirement is that the search [will create a TopScoreDocCollector for each segment](https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java#L404), each one with numHits = 5 million. This is fine for the large segments, but many of those segments are fairly small and only contain several thousand documents. This wastes a huge amount of memory for queries with large values of numHits on indices with many segments.

Therefore, I propose to change the CollectorManager - interface in the following way:
- change the method newCollector to accept a parameter LeafSlice that can be used to determine the total count of documents in the LeafSlice
- Maybe, in order to remain backwards compatible, it would be possible to introduce this as a new method with a default implementation that calls the old method - otherwise, it probably has to wait for Lucene 8?
- This can then be used to cap numHits for each TopScoreDocCollector to the leafslice-size.

If this is something that would make sense for you, I can try to provide a patch.

---
Migrated from [LUCENE-8542](https://issues.apache.org/jira/browse/LUCENE-8542) by Christoph Kaser, updated Mar 13 2019
Attachments: [LUCENE-8542.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8542/LUCENE-8542.patch)

Contributor guide

Open the contributing guide

Research direction

Review IndexSearcher.java around the linked collector creation code and the CollectorManager interface, then inspect the attached LUCENE-8542.patch. Determine how LeafSlice can provide its document count and how the API can remain compatible. Done means small slices no longer allocate collectors for the full requested hit count.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.