apache / apache/lucene

Concordance/Key Word In Context (KWIC) capability [LUCENE-5317]

Open
#6,381 22 comments 0 reactions 1 assignee Claimed by @tteofili View on GitHub
affects-version:4.5 legacy-jira-label:patch legacy-jira-priority:Major module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This patch enables a Lucene-powered concordance search capability.

Concordances are extremely useful for linguists, lawyers and other analysts performing analytic search vs. traditional snippeting/document retrieval tasks. By "analytic search," I mean that the user wants to browse every time a term appears (or at least the topn) in a subset of documents and see the words before and after.

Concordance technology is far simpler and less interesting than IR relevance models/methods, but it can be extremely useful for some use cases.

Traditional concordance sort orders are available (sort on words before the target, words after, target then words before and target then words after).

Under the hood, this is running SpanQuery's getSpans() and reanalyzing to obtain character offsets. There is plenty of room for optimizations and refactoring.

Many thanks to my colleague, Jason Robinson, for input on the design of this patch.

---
Migrated from [LUCENE-5317](https://issues.apache.org/jira/browse/LUCENE-5317) by Tim Allison (@tballison), 4 votes, updated Nov 24 2019
Attachments: [concordance_v1.patch.gz](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5317/concordance_v1.patch.gz), [LUCENE-5317.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5317/LUCENE-5317.patch) (versions: 2), [lucene5317v1.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5317/lucene5317v1.patch), [lucene5317v2.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5317/lucene5317v2.patch)
Linked issues:
- #6382
- [SOLR-5411](https://issues.apache.org/jira/browse/SOLR-5411)
- [SOLR-5412](https://issues.apache.org/jira/browse/SOLR-5412)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.