apache / apache/lucene

Examine the potential of loading vector data directly from the memory segment

Open
#12,482 7 comments 1 reaction 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Just a crazy idea! Thanks @jpountz ;-)

The Panama Vector API supports loading directly from a memory segment. If we could do this, then for vector similarity purposes we could load the underlying vector data directly from the MemorySegment (rather than the current copy from the segment into a float[] ).

A quick experiment, if I did it correctly, and only with float dotProduct, shows approx a 2x performance improvement. ( the experiment builds upon @rmuir 's vectorbench, which we used for the original work that integrated the Panama Vector API )

```
Linux, AVX 512
Benchmark (size) Mode Cnt Score Error Units
FloatDotProductBenchmark.dotProductCopyFromArray 1024 thrpt 5 9.714 ± 0.002 ops/us
FloatDotProductBenchmark.dotProductFromMemorySegment 1024 thrpt 5 18.111 ± 0.008 ops/us
```
https://github.com/ChrisHegarty/vectorbench/blob/5380ebf01df77e1c56d9f392a391df780c19d35e/src/main/java/testing/FloatDotProductBenchmark.java

The benchmark method, `dotProductCopyFromArray`, is what we have in Lucene today, but I added a copy from the memory segment into the two float[]'s to simulate `MemorySegmentIndexInput::readFloats`.

The benchmark method, `dotProductFromMemorySegment`, has the exact same shape of vector operations, just that the values are loaded directly from the memory segment using `FloatVector::fromMemorySegment`. A single segment holds both vectors. I suspect we'd want similar in Lucene too is this were ever to proceed.

In practice I don't see any straightforward way that we could approach this, and there is the complication of using preview / Incubating JDK APIs. That aside, I think that the general idea is sound - avoid the extra intermediate copy to the float[].

I filed this issue mostly to raise awareness of `fromMemorySegment`, and also to provoke a discussion and get thoughts from others. Is it worth trying to pursue? Maybe just crudely hack something and get rough luceneutil bench results to see if / how much it could affect things? `MemorySegmentIndexInput` could expose a way to return XXVector's from a given position? Something else?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked vectorbench experiment, especially FloatDotProductBenchmark.java, then inspect Lucene's MemorySegmentIndexInput and current vector similarity loading path. Run comparable luceneutil benchmarks to measure direct MemorySegment loads against the existing float[] copy. Done means a measured performance result and a clear decision about whether the approach is practical given the preview or incubating JDK APIs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.