PerFieldSimilarityWrapper issue with queryNorm() and coord() [LUCENE-4559]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This issue requests that documentation be clarified for the current
behavior of queryNorm() and coord() on PerFieldAnalyzerWrapper and
that support is added for the use case described below.
The documentation for PerFieldAnalyzerWrapper (lucene 4.0) says:
```
Subclasses should implement get(String) to return an appropriate
Similarity (for example, using field-specific parameter values) for
the field.
```
This is misleading because of the behavior for queryNorm() and
coord(). The Similarity returned from get() is not accessed for these
methods. Instead, the PerFieldAnalyzerWrapper subclass methods are
called. I understand that this is because these methods apply to the
query as a whole rather than per field. However, consider the
following. A PerFieldAnalyzerWrapper with no per-field behavior (just
returns DefaultSimilarity in get()) behaves differently than
DefaultSimilarity itself:
```
class MyPerFieldSimilarity1 extends PerFieldSimilarityWrapper {
`@Override`
public Similarity get(String name) {
return new DefaultSimilarity();
}
}
public class PerFieldSimilarityWrapperTest {
private float runQuery(Similarity similarity) throws IOException {
IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_40, new WhitespaceAnalyzer(Version.LUCENE_40));
config.setSimilarity(similarity);
Directory dir = new RAMDirectory();
IndexWriter writer = new IndexWriter(dir, config);
Document doc = new Document();
doc.add(new TextField("A-field", "first", Store.YES));
writer.addDocument(doc);
writer.commit();
IndexReader reader = DirectoryReader.open(dir);
IndexSearcher searcher = new IndexSearcher(reader);
searcher.setSimilarity(similarity);
TermQuery query = new TermQuery(new Term("A-field", "first"));
TopDocs topDocs = searcher.search(query, 1);
return topDocs.scoreDocs[0].score;
}
`@Test`
public void testSimple() throws Exception {
float score1 = runQuery(new DefaultSimilarity());
float score2 = runQuery(new MyPerFieldSimilarity1());
assertEquals(score1, score2, 0.0001);
// java.lang.AssertionError:
// expected:<0.3068528175354004> but was:<0.09415864944458008>
}
```
One solution is to override and forward, e.g.
```
class MyPerFieldSimilarity1 extends PerFieldSimilarityWrapper {
`@Override`
public Similarity get(String name) {
return new DefaultSimilarity();
}
`@Override`
public float coord(int overlap, int maxOverlap) {
return get("dummy").coord(overlap, maxOverlap);
}
`@Override`
public float queryNorm(float valueForNormalization) {
return get("dummy").queryNorm(valueForNormalization);
}
}
```
However, these methods don't have access to query field data, thus the
"dummy" argument.
Suppose an application arranges documents so that there are two
distinct field groupings:
```
Document:
A-field1
A-field2
A-field3
B-field1
B-field2
B-field3
```
The application creates queries that use the A fields, or the B
fields, but never both A and B in the same query. Then it seems
reasonable that PerFieldAnalyzerWrapper should provide a way for
queryNorm() and coord() to operate on these sets of fields. This
cannot be done with the current implementation.
---
Migrated from [LUCENE-4559](https://issues.apache.org/jira/browse/LUCENE-4559) by Joel Barry
Contributor guide
Research direction
Start with PerFieldSimilarityWrapper and its queryNorm() and coord() methods, then reproduce the score difference shown in PerFieldSimilarityWrapperTest. Review how field-specific Similarity instances are selected and determine how documentation and grouped-field support should behave; done means the documented behavior is accurate and the described A-versus-B query use case is supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100