apache / apache/lucene

Add post-filter capability to `SynonymGraphFilter`

Open
#13,004 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

[I'm not sure how general this is but figured I'd open this to see if there is interest / other use cases:]

At Amazon product search team we have synonyms that are sometimes conditionally applied depending on some context about the document or query. For example, `apple` might be a fruit in the grocery subset of Amazon's catalog, or a computer in the electronics catalog.

Today we implement this very inefficiently: we compile N massive synonym maps, mostly (wastefully) with the same synonyms except for a few that are catalog / query context specific. This is wasteful and takes gobbs of heap. (Hmm, separately: `SynonymGraphFilter` should be fixed to use off-heap FSTs -- I'll open a crab spinoff issue).

Maybe instead we could allow each synonym rule to optionally have some metadata that may be used, at matching time, to post-filter, only applying the synonym based on the context of the current document/query? I'm not sure how this would work -- maybe N labels that are compiled to an int/long bitset recorded into each FST rule?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing SynonymGraphFilter and how its synonym rules are compiled and matched. The issue does not name files or tests, so trace the existing filter entry point and identify how document or query context could reach post-filtering. Done would require a concrete, agreed design for metadata-driven filtering and clear validation criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.