Detect upgrades with non-default formats [LUCENE-9222]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Lucene doesn't give any backward-compatibility guarantees with non-default formats, but doesn't try to detect such misuse either, and a couple users fell in this trap over the years, see e.g. [SOLR-14254](https://issues.apache.org/jira/browse/SOLR-14254).
What about dynamically creating the version number of the index format based on the current Lucene version, so that Lucene would fail with an IndexFormatTooOldException with non-default formats instead of a confusing CorruptIndexException. The change would consist of doing something like that for all our non-default index formats:
```Java
diff --git a/lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsWriter.java b/lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsWriter.java
index fcc0d00a593..18b35760aec 100644
--- a/lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsWriter.java
+++ b/lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsWriter.java
`@@` -41,6 +41,7 `@@` import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.FixedBitSet;
import org.apache.lucene.util.IOUtils;
import org.apache.lucene.util.IntsRefBuilder;
+import org.apache.lucene.util.Version;
import org.apache.lucene.util.fst.FSTCompiler;
import org.apache.lucene.util.fst.FST;
import org.apache.lucene.util.fst.Util;
`@@` -123,7 +124,7 `@@` import org.apache.lucene.util.fst.Util;
public class FSTTermsWriter extends FieldsConsumer {
static final String TERMS_EXTENSION = "tfp";
static final String TERMS_CODEC_NAME = "FSTTerms";
- public static final int TERMS_VERSION_START = 2;
+ public static final int TERMS_VERSION_START = (Version.LATEST.major << 16) | (Version.LATEST.minor << 8) | Version.LATEST.bugfix;
public static final int TERMS_VERSION_CURRENT = TERMS_VERSION_START;
final PostingsWriterBase postingsWriter;
```
---
Migrated from [LUCENE-9222](https://issues.apache.org/jira/browse/LUCENE-9222) by Adrien Grand (@jpountz), updated Feb 24 2020
Linked issues:
- #10288
Contributor guide
Research direction
Start with lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsWriter.java and inspect the other non-default index formats for their version constants. Review Version.LATEST, IndexFormatTooOldException, and CorruptIndexException, then verify that each non-default format derives its version from the current Lucene version and rejects incompatible upgrades with the intended exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100