apache / apache/lucene

Configurable Block Size for Stored Fields

Open
#13,099 2 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

It was observed, for duplicated / similar type of data, the change in block size from [60K to 8K](https://github.com/apache/lucene/commit/f1fdd24) results in over 50% increase in the stored fields size.

This observation is coming from OpenSearch: https://github.com/opensearch-project/OpenSearch/issues/3769.

I was able to replicate the results for duplicated documents: https://github.com/opensearch-project/OpenSearch/issues/3769#issuecomment-1938506593. This also includes the comparison of non-similar data, where the affect of block size is mostly insignificant.

Now currently, if my understanding is correct, there is not a clean way to toggle the block sizes of the codecs without creating a separate Codec and a StoredFieldsFormat (I took a stab at this approach over [here](https://github.com/opensearch-project/OpenSearch/pull/12029)).

I would like to get community's feedback if we could provide a way to make the block size configurable which allows users to choices based on their type of workload.

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked Lucene commit and the OpenSearch issue and pull request, including the stored-fields block-size observations. Determine whether the proposed configurability is accepted and what API or codec design the community wants; the issue currently does not define an implementation path or concrete completion test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.