Examine the potential of loading vector data directly from the memory segment
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Just a crazy idea! Thanks @jpountz ;-)
The Panama Vector API supports loading directly from a memory segment. If we could do this, then for vector similarity purposes we could load the underlying vector data directly from the MemorySegment (rather than the current copy from the segment into a float[] ).
A quick experiment, if I did it correctly, and only with float dotProduct, shows approx a 2x performance improvement. ( the experiment builds upon @rmuir 's vectorbench, which we used for the original work that integrated the Panama Vector API )
```
Linux, AVX 512
Benchmark (size) Mode Cnt Score Error Units
FloatDotProductBenchmark.dotProductCopyFromArray 1024 thrpt 5 9.714 ± 0.002 ops/us
FloatDotProductBenchmark.dotProductFromMemorySegment 1024 thrpt 5 18.111 ± 0.008 ops/us
```
https://github.com/ChrisHegarty/vectorbench/blob/5380ebf01df77e1c56d9f392a391df780c19d35e/src/main/java/testing/FloatDotProductBenchmark.java
The benchmark method, `dotProductCopyFromArray`, is what we have in Lucene today, but I added a copy from the memory segment into the two float[]'s to simulate `MemorySegmentIndexInput::readFloats`.
The benchmark method, `dotProductFromMemorySegment`, has the exact same shape of vector operations, just that the values are loaded directly from the memory segment using `FloatVector::fromMemorySegment`. A single segment holds both vectors. I suspect we'd want similar in Lucene too is this were ever to proceed.
In practice I don't see any straightforward way that we could approach this, and there is the complication of using preview / Incubating JDK APIs. That aside, I think that the general idea is sound - avoid the extra intermediate copy to the float[].
I filed this issue mostly to raise awareness of `fromMemorySegment`, and also to provoke a discussion and get thoughts from others. Is it worth trying to pursue? Maybe just crudely hack something and get rough luceneutil bench results to see if / how much it could affect things? `MemorySegmentIndexInput` could expose a way to return XXVector's from a given position? Something else?
Contributor guide
Research direction
Start by reviewing the linked vectorbench experiment, especially FloatDotProductBenchmark.java, then inspect Lucene's MemorySegmentIndexInput and current vector similarity loading path. Run comparable luceneutil benchmarks to measure direct MemorySegment loads against the existing float[] copy. Done means a measured performance result and a clear decision about whether the approach is practical given the preview or incubating JDK APIs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100