apache / apache/lucene

Optimize the default NormsFormat for the case that all norms are in 0..16 [LUCENE-7839]

Open
#8,890 5 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Given how we now store the length of the field in norms, we could optimize the default norms format for the case that all norms are in 0..16 and store it on 4 bits. This would be picked up for short fields that have less than 16 terms (eg. title fields) and reduce disk utilization by 2.

---
Migrated from [LUCENE-7839](https://issues.apache.org/jira/browse/LUCENE-7839) by Adrien Grand (@jpountz), updated May 24 2017
Attachments: [LUCENE-7839.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7839/LUCENE-7839.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the default norms format and the attached LUCENE-7839.patch referenced in the issue. Determine how norms values in 0..16 could be stored on 4 bits, and consider the short-field case described. Done means reducing disk utilization for applicable norms without changing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.