apache / apache/lucene

Explore moving ByteBuffer(sData|Index)Input to absolute bulk gets [LUCENE-10394]

Open
#11,430 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

With the move to Java 17, we now have access to absolute bulk gets on ByteBuffers: https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/ByteBuffer.html#get(int,byte%5B%5D,int,int). We should look into whether this helps with our more random-access workloads like binary doc values, conjunctive queries and building HNSW graphs.

ByteBuffersDataInput already tries to access the underlying buffers in a random-access fashion and works around the lack of absolute bulk gets by doing `ByteBuffer#duplicate()`. It looks like a low hanging fruit to stop duplicating the buffer and just do an absolute bulk get instead. ByteBuffersIndexInput would require a bit more work since it's performing relative reads whenever possible.

---
Migrated from [LUCENE-10394](https://issues.apache.org/jira/browse/LUCENE-10394) by Adrien Grand (@jpountz), updated Feb 25 2022

Contributor guide

Open the contributing guide

Research direction

Start with ByteBuffersDataInput and ByteBuffersIndexInput, then read the Java 17 ByteBuffer absolute bulk get API. Compare the current duplicate-based random access with the relative-read paths, considering binary doc values, conjunctive queries, and HNSW graph building. Done means the relevant access paths have been evaluated and any beneficial change is validated for correctness and workload performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.