Add post-filter capability to `SynonymGraphFilter`
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
[I'm not sure how general this is but figured I'd open this to see if there is interest / other use cases:]
At Amazon product search team we have synonyms that are sometimes conditionally applied depending on some context about the document or query. For example, `apple` might be a fruit in the grocery subset of Amazon's catalog, or a computer in the electronics catalog.
Today we implement this very inefficiently: we compile N massive synonym maps, mostly (wastefully) with the same synonyms except for a few that are catalog / query context specific. This is wasteful and takes gobbs of heap. (Hmm, separately: `SynonymGraphFilter` should be fixed to use off-heap FSTs -- I'll open a crab spinoff issue).
Maybe instead we could allow each synonym rule to optionally have some metadata that may be used, at matching time, to post-filter, only applying the synonym based on the context of the current document/query? I'm not sure how this would work -- maybe N labels that are compiled to an int/long bitset recorded into each FST rule?
Contributor guide
Research direction
Start by reviewing SynonymGraphFilter and how its synonym rules are compiled and matched. The issue does not name files or tests, so trace the existing filter entry point and identify how document or query context could reach post-filtering. Done would require a concrete, agreed design for metadata-driven filtering and clear validation criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100