apache / apache/lucene

Codec classes/packages that do not provide (automatic) file format back should throw an explicit exception [LUCENE-5126]

Open
#6,190 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

If a user builds an index with an alternate codec that does not provide (automatic) backcompat for reads in future versions of lucene, the resulting error if the user tries to upgrade can be obscure and missleading – suggesting either index corruption or a code bug w/o any obvious indicator that the real problem is that the user needs to explicitly convert the index format. (Even if we document the hell out of this – #6189 – so users have to know about it when they pick a codec, the person who picked the codec for an app may be long gone when some other poor sap tries to upgrade that app to use a newer version of Lucene)

For example, as demonstrated in #6185 if a user uses "Disk" based doc values in 4.2.1, and then tries upgrading to 4.4, they might get...

```
java.io.EOFException: read past EOF: MMapIndexInput(path="/home/hossman/tmp/disk_docvals_bug/collection1/data/index/_0_Disk_0.dvdm")
```

or they might get...

```
org.apache.lucene.index.CorruptIndexException: invalid type: 65, resource=MMapIndexInput(path="/storage/solrdata/index/_fd_Disk_0.dvdm")
```

...it would be a lot better for our users if in situations like this we could give them a clear and definitive indication (similar to "IndexFormatTooOldException") that the codec they are using has changed in a non-backcompat way, and that they need to explicitly convert their index to supported codec before upgrading.

---
Migrated from [LUCENE-5126](https://issues.apache.org/jira/browse/LUCENE-5126) by Chris M. Hostetter (@hossman)

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by tracing how alternate codecs are read during index upgrades, using the Disk doc-values failures in issue #6185 and the related documentation issue #6189 as context. Done means a clear, definitive exception identifies non-backward-compatible codec changes and tells users to convert the index before upgrading.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.