apache / apache/lucene

Detect upgrades with non-default formats [LUCENE-9222]

Open
#10,262 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Lucene doesn't give any backward-compatibility guarantees with non-default formats, but doesn't try to detect such misuse either, and a couple users fell in this trap over the years, see e.g. [SOLR-14254](https://issues.apache.org/jira/browse/SOLR-14254).

What about dynamically creating the version number of the index format based on the current Lucene version, so that Lucene would fail with an IndexFormatTooOldException with non-default formats instead of a confusing CorruptIndexException. The change would consist of doing something like that for all our non-default index formats:

```Java
diff --git a/lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsWriter.java b/lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsWriter.java
index fcc0d00a593..18b35760aec 100644
--- a/lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsWriter.java
+++ b/lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsWriter.java
`@@` -41,6 +41,7 `@@` import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.FixedBitSet;
import org.apache.lucene.util.IOUtils;
import org.apache.lucene.util.IntsRefBuilder;
+import org.apache.lucene.util.Version;
import org.apache.lucene.util.fst.FSTCompiler;
import org.apache.lucene.util.fst.FST;
import org.apache.lucene.util.fst.Util;
`@@` -123,7 +124,7 `@@` import org.apache.lucene.util.fst.Util;
public class FSTTermsWriter extends FieldsConsumer {
static final String TERMS_EXTENSION = "tfp";
static final String TERMS_CODEC_NAME = "FSTTerms";
- public static final int TERMS_VERSION_START = 2;
+ public static final int TERMS_VERSION_START = (Version.LATEST.major << 16) | (Version.LATEST.minor << 8) | Version.LATEST.bugfix;
public static final int TERMS_VERSION_CURRENT = TERMS_VERSION_START;

final PostingsWriterBase postingsWriter;
```

---
Migrated from [LUCENE-9222](https://issues.apache.org/jira/browse/LUCENE-9222) by Adrien Grand (@jpountz), updated Feb 24 2020
Linked issues:
- #10288

Contributor guide

Open the contributing guide

Research direction

Start with lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsWriter.java and inspect the other non-default index formats for their version constants. Review Version.LATEST, IndexFormatTooOldException, and CorruptIndexException, then verify that each non-default format derives its version from the current Lucene version and rejects incompatible upgrades with the intended exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.