apache / apache/lucene

Get field number from the SegmentReadState when get norms

Open
#13,685 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

One of our customers are using Elasticsearch in their own IDC, their cluster has been updated with many versions, including almost all versions from 7.2.0 to 8.9.0, corresponding to Lucene versions 8.0.0 to 9.7.0. Recently, they found that some of the indices would fail to merge. The error stack is as follows:

![image](https://github.com/user-attachments/assets/8881f767-ac7c-4064-ae31-91c471214d8a)

After investigation, it was found that the reason is the inconsistency between the `SegmentReader.fieldInfos` and the `SegmentCoreReader.coreFieldInfos`. The fieldInfo passed to `Lucene80NormsProducer#getNorms` method is from `SegmentReader` and the `Lucene80NormsProducer.norms` is constructed from `SegmentCoreReader.coreFieldInfos`. Currently it is not clear what caused this, but normal querying and reading are not affected. If anyone else in the community has encountered the same problem, welcome to discuss.

Then I imitated the way of getting values from `PointsReader`,that is, getting the field number from the `SegmentReadState`, and found that the merge can be executed successfully.

https://github.com/apache/lucene/blob/4e3945ed548f40f31498dddcdce611c5cbbec245/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90PointsReader.java#L135-L136

I wonder if there is any reason why we don't get field number from `SegmentReadState` for norms just like points. Can we align the way of obtaining field number to avoid some rare error like this(maybe)?

Contributor guide

Open the contributing guide

Research direction

Compare field-number retrieval in Lucene90PointsReader around the referenced lines with Lucene80NormsProducer#getNorms and the SegmentReadState used to construct norms. Trace the mismatch between SegmentReader.fieldInfos and SegmentCoreReader.coreFieldInfos, then verify whether aligning the lookup prevents the reported merge failure without affecting normal querying and reading.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.