Building a Lucene posting format that leverages the Java Vector API
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
This issue is to start a conversation on implementing a vectorized encoding and decoding scheme for postings.
A few months ago, we implemented vectorized integer compression based on the [JavaFastPFOR](https://github.com/lemire/JavaFastPFOR) library. That code has since been [merged](https://github.com/lemire/JavaFastPFOR/pull/51). Performance results, based on [JMH](https://github.com/openjdk/jmh), show [significant gains](https://github.com/mulugetam/VectorJavaFastPFOR) in performance compared to the default JavaFastPFOR.
We would, of course, need to benchmark the vectorized PostingsFormat against the existing implementation.
@jpountz What's your take on it, and how should we go about it?
Contributor guide
Research direction
This is an exploratory proposal rather than a scoped implementation task. Start by reading the existing vectorized integer compression work based on JavaFastPFOR and its JMH performance results, then compare the current PostingsFormat implementation and benchmarking approach. The issue needs an agreed design, implementation scope, and benchmark against the existing format before completion is well defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100