apache / apache/lucene

IndexWriter Initialisation dependent on .fnm files [LUCENE-9993]

Open
#11,032 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-label:IndexWriter legacy-jira-priority:Trivial module:core/index type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I am working on creating an abstraction over Lucene wherein I have 2 places where data is stored: local disk and remote cloud storage. In case the host on which index is present gets terminated due to some issue, I want to be able to replicate the index on another host.

While trying to recreate index on another host, I start by downloading the metadata files associated with the index (segment_N, .si files) and once done with this, I try to initialise an IndexWriter object on top of the local directory to which this has been downloaded from remote storage. This helps me begin indexing the data (I don't have any updateDocs call and only addDocs operation is used) without the need to download data for older segments.

While doing so, I am seeing error during the initialization of IndexWriter itself as it tries to get the [field number mappings](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L1116) for the previous segments before it can create the IndexWriter object.

With the compound file system enabled, this requires to download .cfs files from the remote storage which in turn increases the time required to initialize the IndexWriter, and thus the time before which new host can accept the incoming requests increases resulting in the application rejecting a large number of customer requests.

- Why do we need the fnm files from previous segments while creating the IndexWriter?

- Could you help with a workaround for this to prevent downloading the extra files apart from commit metadata

---
Migrated from [LUCENE-9993](https://issues.apache.org/jira/browse/LUCENE-9993) by Mohit Godwani

Contributor guide

Open the contributing guide

Research direction

Start with the IndexWriter.java field-number-mapping logic linked in the issue, then trace how previous segment metadata and .fnm files are accessed during initialization. Compare this with the local metadata-only replica and compound-file behavior described in the report. Done would require a decided explanation or an explicitly scoped way to avoid downloading the extra files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.