Tighten up IW's CFS codepath [LUCENE-5988]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I wanted to tackle this really in #7031, but I found dragons here.
- the handling of si.files() logic as it relates to compound files is inconsistent. For instance, flush passes trackingdirectorywrapper and has some logic, but merge/addindexes pass the raw directory and track things differently. Ideally we would just use trackingwrapper consistently, and remove CompoundFormat.files().
- merge exception handling is scary: it manually snipes CFS files with indexfiledeleter when exceptions happen, which scares me a lot. I can also "see things that look like bugs" in this code. Maybe we can clean this up (especially if si.files is no longer crazy) and just somehow do a ifd.refresh(newseg) in all cases? Somewhat related is #7049 but this would be a simpler step.
- the timing around setting useCFS boolean is really awkward, e.g. the codec will see false when writing CFS files.
---
Migrated from [LUCENE-5988](https://issues.apache.org/jira/browse/LUCENE-5988) by Robert Muir (@rmuir)
Contributor guide
Research direction
Start by tracing si.files() and CompoundFormat.files() through flush, merge, and addIndexes, comparing TrackingDirectoryWrapper with raw-directory handling. Then inspect merge exception cleanup with IndexFileDeleter, the possible ifd.refresh(newseg) path, and when useCFS is set. Done means the CFS path has consistent tracking, safer exception handling, and correct codec timing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100