apache / apache/lucene

Allow reading binary doc values as a DataInput

Open
#12,459 5 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Binary doc values allow to store a variable number of bytes on a doc value. In order to read those bytes, we currently get a BytesRef from the API which contains the bytes on heap. In order to do that, the current implementation preallocates a byte array with the size equals to the biggest doc value. This strategy has two main drawbacks:

1) We are using a byte array as a middle data structure so we are copying each doc value in this byte array. In many cases this is an unnecessary overhead.

2) when one of the doc values is big, in the order of few megabytes, it can cause issues with small heaps (or even big heaps if big enough). This is due to the allocation of a big byte arrays upfront, that can be consider humongous allocations by the G1 garbage collector and it can cause heap issues under high load.

Therefore I would like to propose to add a new API on top of binary doc values that allows reading them using a DataInput. This data input can read directly from the underlaying IndexInput and therefore we don't need to copy data into an intermediate data structure and we don't need to preallocate a byte array.

The new API would be built on top of the existing binary doc values and it would look something like:

```
public abstract class DataInputDocValues extends DocValuesIterator {

/** Sole constructor. (For invocation by subclass constructors, typically implicit.) */
protected DataInputDocValues() {}

/**
* Returns the binary value wrapped as a {@link DataInput} for the current document ID. It is
* illegal to call this method after {@link #advanceExact(int)} returned {@code false}.
*
* @return the binary value wrapped as a {@link DataInput}
*/
public abstract DataInput dataInputValue() throws IOException;
}
```

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing binary doc values API and the underlying IndexInput path described in the issue. Determine how a DataInput-backed value should behave around advanceExact(int), current-document reads, and IOException handling. Done means the API supports direct binary-value reads without the described intermediate byte-array allocation, with appropriate coverage added.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.