Configurable Block Size for Stored Fields
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
It was observed, for duplicated / similar type of data, the change in block size from [60K to 8K](https://github.com/apache/lucene/commit/f1fdd24) results in over 50% increase in the stored fields size.
This observation is coming from OpenSearch: https://github.com/opensearch-project/OpenSearch/issues/3769.
I was able to replicate the results for duplicated documents: https://github.com/opensearch-project/OpenSearch/issues/3769#issuecomment-1938506593. This also includes the comparison of non-similar data, where the affect of block size is mostly insignificant.
Now currently, if my understanding is correct, there is not a clean way to toggle the block sizes of the codecs without creating a separate Codec and a StoredFieldsFormat (I took a stab at this approach over [here](https://github.com/opensearch-project/OpenSearch/pull/12029)).
I would like to get community's feedback if we could provide a way to make the block size configurable which allows users to choices based on their type of workload.
Contributor guide
Research direction
Start by reading the linked Lucene commit and the OpenSearch issue and pull request, including the stored-fields block-size observations. Determine whether the proposed configurability is accepted and what API or codec design the community wants; the issue currently does not define an implementation path or concrete completion test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100