apache / apache/lucene

Strengthen checks on allowed segment file names [LUCENE-6353]

Open
#7,414 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-fix-version:5.2 legacy-jira-fix-version:6.0 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Replication layers that need to move Lucene indices around frequently need to (or should) be able to decode an given index file into its segment name, segment suffix (generation and/or per-field "suffix"), file name extension.

This is useful for robustness, e.g. copying all of a segment's files if any checksums/ids across those files disagrees between source and replica.

But we don't do very strong checking of the "syntax" of an index file name today, or at least we should do the TODO in SegmentWriteState (upgrade the assert on the segment suffix to a real check).

I think we could also make MDW.close insist every file it finds in a closed index either matches the required syntax, or is an "extra" added by ExtraFS?

---
Migrated from [LUCENE-6353](https://issues.apache.org/jira/browse/LUCENE-6353) by Michael McCandless (@mikemccand), updated May 09 2016

Contributor guide

Open the contributing guide

Research direction

Start with the TODO in SegmentWriteState and inspect MDW.close, focusing on how segment names, suffixes, generations, and extensions are validated. Determine the required filename syntax and how ExtraFS marks additional files; done means closed-index files are rejected unless they match the syntax or are valid extras.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.