Explore moving ByteBuffer(sData|Index)Input to absolute bulk gets [LUCENE-10394]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
With the move to Java 17, we now have access to absolute bulk gets on ByteBuffers: https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/ByteBuffer.html#get(int,byte%5B%5D,int,int). We should look into whether this helps with our more random-access workloads like binary doc values, conjunctive queries and building HNSW graphs.
ByteBuffersDataInput already tries to access the underlying buffers in a random-access fashion and works around the lack of absolute bulk gets by doing `ByteBuffer#duplicate()`. It looks like a low hanging fruit to stop duplicating the buffer and just do an absolute bulk get instead. ByteBuffersIndexInput would require a bit more work since it's performing relative reads whenever possible.
---
Migrated from [LUCENE-10394](https://issues.apache.org/jira/browse/LUCENE-10394) by Adrien Grand (@jpountz), updated Feb 25 2022
Contributor guide
Research direction
Start with ByteBuffersDataInput and ByteBuffersIndexInput, then read the Java 17 ByteBuffer absolute bulk get API. Compare the current duplicate-based random access with the relative-read paths, considering binary doc values, conjunctive queries, and HNSW graph building. Done means the relevant access paths have been evaluated and any beneficial change is validated for correctness and workload performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100