apache / apache/lucene

Co-occurrence filter [LUCENE-2749]

Open
#3,823 8 comments 0 reactions 0 assignees View on GitHub
affects-version:3.1 affects-version:4.0-ALPHA legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The co-occurrence filter to be developed here will output sets of tokens that co-occur within a given window onto a token stream.

These token sets can be ordered either lexically (to allow order-independent matching/counting) or positionally (e.g. sliding windows of positionally ordered co-occurring terms that include all terms in the window are called n-grams or shingles).

The parameters to this filter will be:

- window size: this can be a fixed sequence length, sentence/paragraph context (these will require sentence/paragraph segmentation, which is not in Lucene yet), or over the entire token stream (full field width)
- minimum number of co-occurring terms: >= 2
- maximum number of co-occurring terms: <= window size
- token set ordering (lexical or positional)

One use case for co-occurring token sets is as candidates for collocations.

---
Migrated from [LUCENE-2749](https://issues.apache.org/jira/browse/LUCENE-2749) by Steven Rowe (@sarowe), updated Apr 05 2018
Linked issues:
- #6382

Contributor guide

Open the contributing guide

Research direction

Start by reading this issue and linked issue #6382, then review the existing Lucene token-stream filtering and search components related to co-occurring terms. Define how window size, minimum and maximum term counts, and lexical or positional ordering should work before implementation. Done means the filter emits the specified token sets for each supported window and ordering.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.