apache / apache/lucene

norms are inconsistent with postings [LUCENE-10271]

Open
#11,307 11 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

See discussion: https://github.com/apache/lucene/pull/477

We should not write any norms when there are no postings. And the number of docs which have norms for a field should exactly match the number of docs with postings for the field (getDocCount).

We shouldn't write crazy norms like 0 for docs without tokens, nor should we have crazy query behavior to try to compensate for elasticsearch users that are using the wrong analysis chain.

---
Migrated from [LUCENE-10271](https://issues.apache.org/jira/browse/LUCENE-10271) by Robert Muir (@rmuir)
Attachments: [LUCENE-10271_prototype.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10271/LUCENE-10271_prototype.patch)

Contributor guide

Open the contributing guide

Research direction

Start with the linked discussion at https://github.com/apache/lucene/pull/477 and the attached LUCENE-10271_prototype.patch, then trace the norms and postings handling around getDocCount. Done means fields write no norms without postings, norms-bearing document counts match getDocCount, and zero norms or compensating query behavior are no longer produced.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.