apache / apache/lucene

DisjunctionMaxQuery do not work well when multiple search term+mm+query fields with different fieldType. [LUCENE-8767]

Open
#9,812 2 comments 0 reactions 0 assignees View on GitHub
affects-version:7.3 legacy-jira-label:patch legacy-jira-priority:Critical module:core/queryparser type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

When multiple fields in query fields came from different fieldType, especially one from KeywordTokenizerFactory, another from WhitespaceTokenizerFactory, then the generated parse query could not honor synonyms and mm, which hit incorrect documents. The following is my detail:
1. We use Solr 7.3.1
1. Our qf=name^10 partNumber_ntk, while fieldType of name use solr.WhitespaceTokenizerFactory and solr.WordDelimiterFilterFactory, while  partNumber_ntk is not tokenized and use solr.KeywordTokenizerFactory
1. mm=2<3 4<5 6<-80%25
1. The search term is versatil sundress, while 'versatile' and 'testing' are synonyms, we have documents named " Versatil Empire Waist Sundress" which should be hit, but failed.
1. We test same query on Solr 5.5.4, it works fine, it do not work on Solr 7.3.1.

q=

(Versatil%20testing)%20sundress&fl=name&defType=edismax&mm=2<3 4<5 6<-80%25&qf=name^10%20partNumber_ntk&debugQuery=true&wt=xml&rows=100

parsedQuery:

+(DisjunctionMaxQuery((((name:versatil name:test)\~2)^10.0 | partNumber_ntk:versatil testing)) DisjunctionMaxQuery(((name:sundress)^10.0 | partNumber_ntk:sundress)))\~2

Which seems it incorrect parse name to: name:versatil name:test

If I change the query fields to same fieldType, for example,shortDescription is in same fieldType of name:

q=(Versatil%20testing)%20sundress&fl=name&defType=edismax&mm=2<3 4<5 6<-80%25&qf=name^10%20shortDescription&debugQuery=true&wt=xml&rows=100

ParsedQuery:

+((DisjunctionMaxQuery(((name:versatil)^10.0 | shortDescription:versatil)) DisjunctionMaxQuery(((name:test)^10.0 | shortDescription:test))) DisjunctionMaxQuery(((name:sundress)^10.0 | shortDescription:sundress)))\~2

which hits correctly.

Could someone check this or tell us a quick workaround? Now it have big impact on customer.

Thanks in advance! The following is backup information:

---
Migrated from [LUCENE-8767](https://issues.apache.org/jira/browse/LUCENE-8767) by ZhongHua Wu, updated Aug 09 2019
Environment:
```
Solr: 7.3.1

Backup:

FieldType for name field:









FieldType for partNumber field:







```

Attachments: [a.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8767/a.diff)

Contributor guide

Open the contributing guide

Research direction

Reproduce the supplied eDisMax query using the documented Solr 7.3.1 field types and inspect the generated DisjunctionMaxQuery, especially handling of synonyms and mm across KeywordTokenizerFactory and WhitespaceTokenizerFactory fields. Done means the parsed query honors synonyms and minimum-should-match consistently for differing field types and no longer misses the described documents.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.