Adding field w/ norms should fail if same field was added w/o norms already [LUCENE-3153]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
A spinoff from #4219. Consider the following two scenarios, according to how 4.0 currently works:
- Field "a" is added w/ norms. Sometime later field "a" is added to a document w/o norms – norms are disabled for field "a", for all docs.
- Field "a" is added w/o norms - norms are disabled for field "a". Sometime later field "a" is added to a document w/ norms – app thinks norms were added, while in fact they are dropped.
This is a bug and case `#2` should fail on add/updateDocument - app should know norms were not added. While case `#1` isn't great either, it's the only way an app can choose to disable norms for field "a", after instances of it already contain norms, so we should support that scenario.
In order to detect that early, we should track norms info in .fnx, as Mike describes at #4219. Since this changes the index format, we should also update the "file format" page after we do it.
Not sure what's the deal w/ 3.x indexes that are read by 4.0 code. Initially they won't have .fnx file, so no central norms information exist to detect the cases I've described above. Over time, as segments are merged, .fnx will include information from more and more segments, but there's always a chance few segments will still contain the norms for field "a". I'm not very familiar w/ that part of the code, but I think that:
- If .fnx says "no norms for field a", the we ignore any norms information that may or may not exist in segments.
- If .fnx says "norms for field a", then we need to make up some norms values for (old) segments w/ no norms? We need to make up values during segment merge and search?
---
Migrated from [LUCENE-3153](https://issues.apache.org/jira/browse/LUCENE-3153) by Shai Erera (@shaie), updated May 09 2016
Contributor guide
Research direction
Start by tracing add/updateDocument handling for field norms and the .fnx index-format changes described in #4219. Review how 3.x indexes are handled, then update the file-format page. Done means adding a field with norms after it was added without norms fails, while the supported reverse scenario remains possible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100