CommonGramsQueryFilter: javadoc example does not produce output indicated [LUCENE-10007]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
CommonGramsQueryFilter has the following explanation of it's behavior + example twice in it's javadocs (both at the class level and the incrementToken method level)..
```
/**
* Wrap a CommonGramsFilter optimizing phrase queries by only returning single words when they are
* not a member of a bigram.
*
*
Example:
*
*
- query input to CommonGramsFilter: "the rain in spain falls mainly"
* - output of CommomGramsFilter/input to CommonGramsQueryFilter: |"the, "the-rain"|"rain"
* "rain-in"|"in, "in-spain"|"spain"|"falls"|"mainly"
* - output of CommonGramsQueryFilter:"the-rain", "rain-in" ,"in-spain", "falls", "mainly"
*
*
*/
```
But this input doesn't actually produce the documented output: CommonGramsQueryFilter does in fact produce a token for "spain" even though it is part of the "in-spain" bigram.
I'm not really sure which is "wrong" – the implementation or the documentation – but something seems to be out of whack.
----
(a more trivial discrepancy is the use of `"_"` [underscore] in the CommomGramsFilter impl vs `"-"` [dash] in the javadoc example)
---
Migrated from [LUCENE-10007](https://issues.apache.org/jira/browse/LUCENE-10007) by Chris M. Hostetter (@hossman), updated Jun 18 2021
Attachments: [LUCENE-10007.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10007/LUCENE-10007.patch)
Linked issues:
- [SOLR-15487](https://issues.apache.org/jira/browse/SOLR-15487)
Contributor guide
Research direction
Start with the CommonGramsQueryFilter class-level and incrementToken Javadocs, then inspect the behavior for the documented "the rain in spain falls mainly" example and compare it with CommonGramsFilter. Determine whether the implementation or documentation is authoritative, then make the duplicated example and underscore/dash terminology consistent with the verified behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100