low level Field metadata is never removed from index [LUCENE-1761]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
with heterogeneous docs, or an index whose fields evolve over time, field names that are no longer used (ie: all docs that ever referenced them have been deleted) still show up when you use IndexReader.getFieldNames.
It seems logical that segment merging should only preserve metadata about fields that actually existing the new segment, but even after deleting all documents from an index and optimizing the old field names are still present.
---
Migrated from [LUCENE-1761](https://issues.apache.org/jira/browse/LUCENE-1761) by Chris M. Hostetter (@hossman), 1 vote, updated Aug 18 2017
Attachments: [LUCENE-1761.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1761/LUCENE-1761.patch)
Contributor guide
Research direction
Start at IndexReader.getFieldNames and trace how segment merging and optimization preserve field metadata. Reproduce the case with heterogeneous or deleted documents, then verify that field names no longer present in the remaining documents are absent after merging or optimizing. The payload names no source files or tests, so locating the relevant merge metadata code is part of the work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100