apache / apache/lucene

Bulk operations for LongValues and Sorted[Set]DocValues [LUCENE-8178]

Open
#9,226 3 comments 0 reactions 0 assignees View on GitHub
affects-version:7.2.1 legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

One-by-one DocValues iteration by `advanceExact` and `nextOrd`/`ordValue` is really slow for bulk operations like facetting. Reading and unpacking integers in blocks is substantially faster but DocValues for now can be queried only for single document.

To apply document-based bulk processing `DocIdSetIterator` matches have to be splitted to sequential docID runs and remapped to underlying `LongValues` positions.
After this transformation relatively large linear scans can be performed over packed integers.

 

To do this two new interfaces

1. `LongValuesCollector` (`collectValue(long index, long value)`).
2. `OrdStatsCollector` (`collectOrd(long ord)`, `collectMissing(int count)`).

and three new functions are introduced

1. `LongValues.forRange(long begin, long end, LongValuesCollector collector)`
2. `SortedDocValues.forEach(DocIdSetIterator disi, OrdStatsConsumer collector)`
3. `SortedSetDocValues.forEach(DocIdSetIterator disi, OrdStatsConsumer collector)`

with reference implementations.

Optimized versions of these functions are provided for:
1. `DirectReader` for non-32/64 bits per value cases (using `PackedInts.Decoder`).
2. `Lucene70DocValuesProducer` `getSorted` and `getSortedSet` (both sparse and dense).

 

Measured Solr facetting performance boost is up to 2 - 2.5x on real index.
Patch for Solr `DocValuesFacets` is also provided as separate file.

 

Implementation notes:
- `OrdStatsCollector` does not accept document id because it will ruin performance for `SortedSetDocValues` due to excessive position lookups.
- This patch is fully compatible with Lucene 7.0 DocValues format.

![graph.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8178/graph.png)

---
Migrated from [LUCENE-8178](https://issues.apache.org/jira/browse/LUCENE-8178) by Nikolay Khitrin (@khitrin), 2 votes, updated Feb 20 2018
Attachments: [graph.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8178/graph.png), [LUCENE-8178.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8178/LUCENE-8178.patch), [LUCENE-8178-for-solr.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8178/LUCENE-8178-for-solr.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reading the attached LUCENE-8178.patch and the existing LongValues, SortedDocValues, SortedSetDocValues, DirectReader, and Lucene70DocValuesProducer implementations. Compare the reference and optimized bulk-processing APIs described in the issue, then verify that the requested collectors and forEach/forRange operations are implemented while preserving Lucene 7.0 DocValues compatibility and the stated performance goal.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.