apache / apache/lucene

Indexing method for learned sparse retrieval

Open
#11,799 10 comments 6 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Recent learned sparse retrieval methods ([Splade](https://github.com/naver/splade), [uniCOIL](https://github.com/castorini/pyserini/blob/master/docs/experiments-unicoil.md)) were trained to generate impact score directly (replacing tf-idf score).
For each document, they will generate a json file with terms and weights, e.g. `{";": 80, "the": 161, "of": 85, "and": 27, "to": 24, "was": 47, "as": 27, "their": 96, "what": 40, "over": 123, "only": 123, "important": 186, "project": 208, "success": 215, "meant": 131, "lives": 140, "presence": 180, "scientific": 200, "communication": 235, "thousands": 142, "hundreds": 144, "truly": 170, "hanging": 141, "cloud": 187, "engineers": 127, "achievement": 192, "researchers": 137, "innocent": 181, "manhattan": 244, "impressive": 191, "equally": 163, "##rated": 132, "minds": 137, "atomic": 214, "amid": 201, "##lite": 120, "intellect": 202, "ob": 140}}`
Can we make a new feature that could index this type of document efficiently?
The current [work-around ](https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/collection/JsonVectorCollection.java) I am aware of is to create a fake document by repeating the terms: e.g., `"the the the the .... of of of of of "`
However, this way is not very efficient if the impact score gets bigger and also it requires impact score quantization before indexing.
I think it would be very useful for many people if we can index the json files directly with float impact scores.

Contributor guide

Open the contributing guide

Research direction

Start by reading JsonVectorCollection.java, the linked workaround, and the issue's Splade and uniCOIL examples to understand the current input path. Done means Lucene can index the described JSON term-weight documents directly with float impact scores, without repeated terms or impact-score quantization.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.