norms are inconsistent with postings [LUCENE-10271]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
See discussion: https://github.com/apache/lucene/pull/477
We should not write any norms when there are no postings. And the number of docs which have norms for a field should exactly match the number of docs with postings for the field (getDocCount).
We shouldn't write crazy norms like 0 for docs without tokens, nor should we have crazy query behavior to try to compensate for elasticsearch users that are using the wrong analysis chain.
---
Migrated from [LUCENE-10271](https://issues.apache.org/jira/browse/LUCENE-10271) by Robert Muir (@rmuir)
Attachments: [LUCENE-10271_prototype.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10271/LUCENE-10271_prototype.patch)
Contributor guide
Research direction
Start with the linked discussion at https://github.com/apache/lucene/pull/477 and the attached LUCENE-10271_prototype.patch, then trace the norms and postings handling around getDocCount. Done means fields write no norms without postings, norms-bearing document counts match getDocCount, and zero norms or compensating query behavior are no longer produced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100