apache / apache/lucene

low level Field metadata is never removed from index [LUCENE-1761]

Open
#2,835 7 comments 0 reactions 0 assignees View on GitHub
affects-version:2.2 affects-version:2.3 affects-version:2.3.1 affects-version:2.3.2 affects-version:2.4 affects-version:2.4.1 legacy-jira-label:gsoc2014 legacy-jira-priority:Minor module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

with heterogeneous docs, or an index whose fields evolve over time, field names that are no longer used (ie: all docs that ever referenced them have been deleted) still show up when you use IndexReader.getFieldNames.

It seems logical that segment merging should only preserve metadata about fields that actually existing the new segment, but even after deleting all documents from an index and optimizing the old field names are still present.

---
Migrated from [LUCENE-1761](https://issues.apache.org/jira/browse/LUCENE-1761) by Chris M. Hostetter (@hossman), 1 vote, updated Aug 18 2017
Attachments: [LUCENE-1761.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1761/LUCENE-1761.patch)

Contributor guide

Open the contributing guide

Research direction

Start at IndexReader.getFieldNames and trace how segment merging and optimization preserve field metadata. Reproduce the case with heterogeneous or deleted documents, then verify that field names no longer present in the remaining documents are absent after merging or optimizing. The payload names no source files or tests, so locating the relevant merge metadata code is part of the work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.