apache / apache/lucene

Add support for subsets of searchables inside a MultiSearcher/ParallelMultiSearcher instance's methods at runtime [LUCENE-2447]

Open
#3,521 10 comments 0 reactions 0 assignees View on GitHub
affects-version:3.0.1 legacy-jira-label:multisearcher legacy-jira-label:parallelmultisearcher legacy-jira-priority:Minor module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Here's the situation: We have a site with a fair few amount of indexes that we're using MultiSearcher/ParallelMultiSearcher for, but the users can select an arbitrary permutation of indexes to search. For example (contrived, but illustratory): the site has indexes numbered 1 - 10; user A wants to search in all 10; user B wants to search indexes 1, 2 and 3, user C wants to search even-numbered indexes. From Lucene 3.0.1, the only way to do this is to continually instantiate a new MultiSearcher based on every permutation of indexes that a user wants, which is not ideal at all.

What I've done is add a new parameter to all methods in MultiSearcher that use the searchables array (docFreq, search, rewrite and createDocFrequencyMap), a Set<Searchable> which is checked for isEmpty() and contains() for every iteration over the searchables[]. The actual logic has been moved into these methods and the old methods have become overloads that pass a Collections.emptySet() into those methods, so I do not expect there to be a very noticeable performance impact as a result of this modification, if it's measurable at all.

I didn't modify the test for MultiSearcher very much, just enough to illustrate the that subsetting of the search results works, since no other logic has changed. If I need to do more for the testing, let me know and I'll do it.

I've attached the patches for MultiSearcher.java, ParallelMultiSearcher.java and TestMultiSearcher.java.

---
Migrated from [LUCENE-2447](https://issues.apache.org/jira/browse/LUCENE-2447) by Edward Drapkin, updated May 06 2010
Environment:
```
Irrelevant
```

Attachments: [LUCENE-2447.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2447/LUCENE-2447.patch), [LUCENE-2447-predicate.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2447/LUCENE-2447-predicate.patch)

Contributor guide

Open the contributing guide

Research direction

Start with MultiSearcher.java and ParallelMultiSearcher.java, then review TestMultiSearcher.java and the attached LUCENE-2447 patches. Run the existing MultiSearcher tests first and compare the proposed runtime subset behavior with the current overloads. Done means supported methods can search a selected subset of searchables while existing behavior remains covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.