apache / apache/lucene

Add support for slow filters with batch processing [LUCENE-2362]

Open
#3,438 7 comments 0 reactions 0 assignees View on GitHub
affects-version:3.0.1 legacy-jira-label:batch legacy-jira-label:filter legacy-jira-label:perfomance legacy-jira-label:search legacy-jira-priority:Major module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Internal implementation of IndexSearch assumes that Filter and scorer has almost equal perfomance. But in our environment we have Filter implementation that is very expensive (in compare to scorer).

if we have, let's say, 2k of termdocs selected by scorer (each \~250 docs) and 2k selected by filter, then 250k docs will be fastly checked (and filtered out) by scorer, and 250k docs will be slowly checked by our filter.

Using straigthforward implementation makes search out of 60 seconds per query boundary, because each next() or advance() requires N queries to database PER CHECKED DOC. Using read ahead technique allows us to optimze it to 35 seconds per query. Still too slow.

The solution to problem is firstly select all documents by scorer and filter them in batch by our filter. Example of implementation (with BitSet) in attachement. Currently it takes only \~300 millseconds per query.

---
Migrated from [LUCENE-2362](https://issues.apache.org/jira/browse/LUCENE-2362) by Sergey Vladimirov, updated Apr 03 2010
Attachments: [BatchFilter.java](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2362/BatchFilter.java), [IndexSearcherImpl.java](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2362/IndexSearcherImpl.java)

Contributor guide

Open the contributing guide

Research direction

Start by reading the attached BatchFilter.java and IndexSearcherImpl.java, then trace how IndexSearch currently combines the scorer and filter. The requested behavior is batch filtering of scorer-selected documents rather than one database query per checked document; done means preserving search results while substantially reducing filter-query overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.