apache / apache/lucene

CommonGramsQueryFilter: javadoc example does not produce output indicated [LUCENE-10007]

Open
#11,046 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

CommonGramsQueryFilter has the following explanation of it's behavior + example twice in it's javadocs (both at the class level and the incrementToken method level)..
```
/**
* Wrap a CommonGramsFilter optimizing phrase queries by only returning single words when they are
* not a member of a bigram.
*
*

Example:
*
*


    *
  • query input to CommonGramsFilter: "the rain in spain falls mainly"
    *
  • output of CommomGramsFilter/input to CommonGramsQueryFilter: |"the, "the-rain"|"rain"
    * "rain-in"|"in, "in-spain"|"spain"|"falls"|"mainly"
    *
  • output of CommonGramsQueryFilter:"the-rain", "rain-in" ,"in-spain", "falls", "mainly"
    *

*/
```
But this input doesn't actually produce the documented output: CommonGramsQueryFilter does in fact produce a token for "spain" even though it is part of the "in-spain" bigram.

I'm not really sure which is "wrong" – the implementation or the documentation – but something seems to be out of whack.

 ----

(a more trivial discrepancy is the use of  `"_"` [underscore] in the CommomGramsFilter impl vs `"-"` [dash] in the javadoc example)

---
Migrated from [LUCENE-10007](https://issues.apache.org/jira/browse/LUCENE-10007) by Chris M. Hostetter (@hossman), updated Jun 18 2021
Attachments: [LUCENE-10007.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10007/LUCENE-10007.patch)
Linked issues:
- [SOLR-15487](https://issues.apache.org/jira/browse/SOLR-15487)

Contributor guide

Open the contributing guide

Research direction

Start with the CommonGramsQueryFilter class-level and incrementToken Javadocs, then inspect the behavior for the documented "the rain in spain falls mainly" example and compare it with CommonGramsFilter. Determine whether the implementation or documentation is authoritative, then make the duplicated example and underscore/dash terminology consistent with the verified behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.