apache / apache/lucene

OfflineSorter shouldn't always forceMerge in the end [LUCENE-7141]

Open
#8,196 4 comments 0 reactions 1 assignee Claimed by @mikemccand View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Today it always does a final merge, to collapse all segments into a single segment.

But typically the caller is going to re-iterate all values anyway, to go off and build an FST or a BKD tree or something, and so that final forceMerge is often not necessary and the merging can be done on the fly when the caller consumes the result.

This is somewhat tricky to do ... I'd like to break it into steps, starting with fixing the `ByteSequencesReader` API to implement `BytesRefIterator` instead of its own `read(BytesRefBuilder)` method as a first step.

---
Migrated from [LUCENE-7141](https://issues.apache.org/jira/browse/LUCENE-7141) by Michael McCandless (@mikemccand), updated Mar 26 2016
Attachments: [LUCENE-7141.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7141/LUCENE-7141.patch)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.