apache / apache/lucene

Separate index/store document APIs take 2?

Open
#12,142 6 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

As @rmuir managed to make me look into reducing the amount of guessing we're doing in our document API, I think that a requirement for doing it right will be to split our index and store document APIs. Currently, `Document` and `IndexableField` are trying to cover both, which creates a confusing API.

For the store API, I wonder if we still need a Document abstraction. Maybe we could return a simple `List` and push the work on users to convert it into a map if they want to be able to access fields by name. This is something that is easy to do with Java streams nowadays.

For the index API, I'd like to model `IndexableField` according to how it's getting consumed by `IndexingChain`. Something like:

```java
class IndexableField {

// Populates the inverted index, IndexableValue wraps one of a BytesRef or a TokenStream
IndexableValue invertedValue(Analyzer analyzer, TokenStream reuse);

// Populates points
byte[] pointValue();

// Populates NUMERIC or SORTED_NUMERIC doc values
long numericDocValue();

// Populates BINARY, SORTED or SORTED_SET doc values
BytesRef docValue();

// Populates stored fields
StoredValue storedValue();

// Populates byte[] vectors
byte[] binaryVectorValue();

// Populates float[] vectors
float[] floatVectorValue();

}
```

I haven't thought much about it but I'm curious to hear thoughts.

Contributor guide

Open the contributing guide

Research direction

Start by reading the current Document and IndexableField APIs alongside their use in IndexingChain. Review the proposed split between index and store APIs and the listed value methods. Done would require an agreed API design and defined changes, which this issue does not yet specify.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.