apache / apache/lucene

Can IW sort segments on flush too? [LUCENE-7281]

Open
#8,336 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Spinoff from #7824, where we made index-time sorting first class in Lucene, except only merged segments get sorted.

Can we also sort flushed segments? This would be cleaner because then search-time users don't need to fret about whether a given segment is sorted or not, and it would also improve the current "best effort" check IW does that you didn't try to change the index sort order, to an accurate check.

But this is tricky because e.g. at least term vectors and stored fields write "live" to the segment's (codec's) files as documents are being indexed...

---
Migrated from [LUCENE-7281](https://issues.apache.org/jira/browse/LUCENE-7281) by Michael McCandless (@mikemccand)

Contributor guide

Open the contributing guide

Research direction

Start by studying the existing index-time sorting and segment-merge behavior referenced in the issue, then trace how flushed segments write term vectors and stored fields directly through codec files. Done means flushed segments are sorted as well as merged segments, and the index-sort check accurately detects attempts to change the index sort order.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.