apache / apache/lucene

GraphTokenStreamFiniteStrings does not handle position inc > 1 in multi-word synoyms [LUCENE-8137]

Open
#9,185 6 comments 0 reactions 1 assignee Claimed by @jimczi View on GitHub
affects-version:7.2.1 affects-version:8.0 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The automaton built for graph queries that contain multiple multi-word synonyms does not handle gaps if they appear in the middle of a multi-word synonym. In such case the token next to the gap is considered as part of the multi-word synonym. 

Stop words that appear before or after multi-word synonyms are handled correctly in the current version but the synonym rule "part of speech, pos" for instance does not create the expected query if "of" is removed by a filter that is set after the synonym_graph.  One solution would be to reuse TokenStreamToAutomaton (with minor changes to add the ability to create token transitions rather than chars) which preserves gaps (as a transition) in the produced automaton.

---
Migrated from [LUCENE-8137](https://issues.apache.org/jira/browse/LUCENE-8137) by Jim Ferenczi (@jimczi), 3 votes, updated Oct 10 2019
Attachments: [SGF_SF_interaction.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8137/SGF_SF_interaction.patch)
Linked issues:
- [SOLR-11968](https://issues.apache.org/jira/browse/SOLR-11968)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.