apache / apache/lucene

Monitor alternative "TermFilter" Presearcher for sparse filter fields [LUCENE-10571]

Open
#11,607 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:monitor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

One of the things that surprised me the most when looking into how the `TermFilteredPresearcher` worked was what happens when Queries and/or Documents do _NOT_  have a value in a configured filter field.

per the javadocs...
> Filtering by additional fields can be configured by passing a set of field names\. Documents that contain values in those fields will only be checked against \{`@link` MonitorQuery} instances that have the same fieldname-value mapping in their metadata\.

...which is straightforward and useful in the tested example where every registered Query has `"language"` metadata, and every Document has a `"language"` field, but gives unintuitive results when a Query or Document does **NOT** have a `"language"`

A more "intuitive" & useful (in my opinions) implementation would be something that could be documented as ...
> Filtering by additional fields can be configured by passing a set of field names\. Documents that contain values in those fields will only be checked against \{`@link` MonitorQuery} instances
> that have the same fieldname-value mapping in their metadata <em>or have no mapping for that fieldname</em>\.
>
> Documents that do not contain values in those fields will only be checked against \{`@link` MonitorQuery} instances that also have no mapping for that fieldname\.

...ie: instead of being a straight "filter candidate queries by what we find in the filter fields in the documents" we can instead "derive the queries that are viable candidates for each document if we were restricting the set of documents by those values during a "forward search"

---
Migrated from [LUCENE-10571](https://issues.apache.org/jira/browse/LUCENE-10571) by Chris M. Hostetter (@hossman), updated Jun 27 2022
Attachments: [LUCENE-10571.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10571/LUCENE-10571.patch)

Contributor guide

Open the contributing guide

Research direction

Start with the issue description and the archived LUCENE-10571.patch, then inspect the TermFilteredPresearcher behavior described in the javadocs. Establish how missing filter values currently behave and define tests covering absent query and document metadata. Done means the intended candidate-selection behavior is implemented and documented with passing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.