apache / apache/lucene

org.apache.lucene.index.IndexFormatTooNewException on arm64

Open
#13,452 3 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

I am trying to run Apache Pinot on ARM64 (Graviton). While the table partitions are loaded successfully on x86, I get the following exception on ARM:

```
org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource ): 9 (needs to be between 6 and 8)
at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:213) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:193) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
at org.apache.lucene.util.fst.FST.(FST.java:454) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
at org.apache.pinot.segment.local.segment.index.readers.LuceneFSTIndexReader.(LuceneFSTIndexReader.java:55) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
at org.apache.pinot.segment.local.segment.index.fst.FstIndexType$ReaderFactory.createIndexReader(FstIndexType.java:200) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
at org.apache.pinot.segment.local.segment.index.fst.FstIndexType$ReaderFactory.createIndexReader(FstIndexType.java:208) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
at org.apache.pinot.segment.local.segment.index.fst.FstIndexType$ReaderFactory.createIndexReader(FstIndexType.java:183) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
at org.apache.pinot.segment.spi.index.IndexReaderFactory$Default.createIndexReader(IndexReaderFactory.java:67) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
at org.apache.pinot.segment.local.segment.index.column.PhysicalColumnIndexContainer.(PhysicalColumnIndexContainer.java:59) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
at
```

While I am still in the process of debugging this, I created this issue in case I can get some help faster. Thanks!

### Version and environment details

Linux/arm64
Lucene version: 9.10.0

Contributor guide

Open the contributing guide

Research direction

Start with the reported stack trace in LuceneFSTIndexReader.java and CodecUtil.java, then inspect the Lucene 9.10.0 ARM64 environment and the index that triggers the format check. Reproduce the failure against the affected Apache Pinot table partitions; done requires identifying a confirmed cause and a reproducible, validated fix or compatibility decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.