Co-occurrence filter [LUCENE-2749]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
The co-occurrence filter to be developed here will output sets of tokens that co-occur within a given window onto a token stream.
These token sets can be ordered either lexically (to allow order-independent matching/counting) or positionally (e.g. sliding windows of positionally ordered co-occurring terms that include all terms in the window are called n-grams or shingles).
The parameters to this filter will be:
- window size: this can be a fixed sequence length, sentence/paragraph context (these will require sentence/paragraph segmentation, which is not in Lucene yet), or over the entire token stream (full field width)
- minimum number of co-occurring terms: >= 2
- maximum number of co-occurring terms: <= window size
- token set ordering (lexical or positional)
One use case for co-occurring token sets is as candidates for collocations.
---
Migrated from [LUCENE-2749](https://issues.apache.org/jira/browse/LUCENE-2749) by Steven Rowe (@sarowe), updated Apr 05 2018
Linked issues:
- #6382
Contributor guide
Research direction
Start by reading this issue and linked issue #6382, then review the existing Lucene token-stream filtering and search components related to co-occurring terms. Define how window size, minimum and maximum term counts, and lexical or positional ordering should work before implementation. Done means the filter emits the specified token sets for each supported window and ordering.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100