apache / apache/lucene

IndexableField changes its IndexableFieldType when the index is re-opened for reading [LUCENE-7171]

Open
#8,226 5 comments 0 reactions 0 assignees View on GitHub
affects-version:5.5 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This code:

```Java
/* Store one document into an index */
Directory index = new RAMDirectory();
IndexWriterConfig config = new IndexWriterConfig(new StandardAnalyzer());
IndexWriter w = new IndexWriter(index, config);

Document d1 = new Document();
d1.add(new StringField("isbn", "9900333X", Field.Store.YES));
w.addDocument(d1);
w.commit();
w.close();

/* inspect IndexableFieldType */
IndexableField f1 = d1.getField("isbn");
System.err.println("FieldType for " + f1.stringValue() + " : " + f1.fieldType());

/* retrieve all documents and inspect IndexableFieldType */
IndexSearcher s = new IndexSearcher(DirectoryReader.open(index));
TopDocs td = s.search(new MatchAllDocsQuery(), 1);

for (ScoreDoc sd : td.scoreDocs) {
Document d2 = s.doc(sd.doc);
IndexableField f2 = d2.getField("isbn");
System.err.println("FieldType for " + f2.stringValue() + " : " + f2.fieldType());
}
```

Produces:

```Java
FieldType for 9900333X : stored,indexed,omitNorms,indexOptions=DOCS
FieldType for 9900333X : stored,indexed,tokenized,omitNorms,indexOptions=DOCS
```

The `StringField` field `isbn` is not tokenized, as correctly reported by the first output, which happens right after closing the writer.
However, it becomes tokenized when the index is re-opened with a new reader.

---
Migrated from [LUCENE-7171](https://issues.apache.org/jira/browse/LUCENE-7171) by Roberto Cornacchia, updated May 23 2016

Contributor guide

Open the contributing guide

Research direction

Start by running the Java reproduction in the issue and compare the IndexableFieldType before and after opening the index with a reader. Trace how the StringField is reconstructed during document retrieval; done means the field type remains consistent and the reproduction reports the expected non-tokenized type.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.