apache / apache/lucene

Tighten up IW's CFS codepath [LUCENE-5988]

Open
#7,050 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I wanted to tackle this really in #7031, but I found dragons here.

- the handling of si.files() logic as it relates to compound files is inconsistent. For instance, flush passes trackingdirectorywrapper and has some logic, but merge/addindexes pass the raw directory and track things differently. Ideally we would just use trackingwrapper consistently, and remove CompoundFormat.files().
- merge exception handling is scary: it manually snipes CFS files with indexfiledeleter when exceptions happen, which scares me a lot. I can also "see things that look like bugs" in this code. Maybe we can clean this up (especially if si.files is no longer crazy) and just somehow do a ifd.refresh(newseg) in all cases? Somewhat related is #7049 but this would be a simpler step.
- the timing around setting useCFS boolean is really awkward, e.g. the codec will see false when writing CFS files.

---
Migrated from [LUCENE-5988](https://issues.apache.org/jira/browse/LUCENE-5988) by Robert Muir (@rmuir)

Contributor guide

Open the contributing guide

Research direction

Start by tracing si.files() and CompoundFormat.files() through flush, merge, and addIndexes, comparing TrackingDirectoryWrapper with raw-directory handling. Then inspect merge exception cleanup with IndexFileDeleter, the possible ifd.refresh(newseg) path, and when useCFS is set. Done means the CFS path has consistent tracking, safer exception handling, and correct codec timing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.