apache / apache/lucene

Building a Lucene posting format that leverages the Java Vector API

Open
#12,090 1 comment 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

This issue is to start a conversation on implementing a vectorized encoding and decoding scheme for postings.

A few months ago, we implemented vectorized integer compression based on the [JavaFastPFOR](https://github.com/lemire/JavaFastPFOR) library. That code has since been [merged](https://github.com/lemire/JavaFastPFOR/pull/51). Performance results, based on [JMH](https://github.com/openjdk/jmh), show [significant gains](https://github.com/mulugetam/VectorJavaFastPFOR) in performance compared to the default JavaFastPFOR.

We would, of course, need to benchmark the vectorized PostingsFormat against the existing implementation.

@jpountz What's your take on it, and how should we go about it?

Contributor guide

Open the contributing guide

Research direction

This is an exploratory proposal rather than a scoped implementation task. Start by reading the existing vectorized integer compression work based on JavaFastPFOR and its JMH performance results, then compare the current PostingsFormat implementation and benchmarking approach. The issue needs an agreed design, implementation scope, and benchmark against the existing format before completion is well defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.