apache / apache/lucene

Multi-token post-analysis DocValues [LUCENE-10023]

Open
#11,062 16 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:core/index type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The single-token case for post-analysis DocValues is accounted for by `Analyzer.normalize(...)` (and formerly `MultiTermAwareComponent`); but there are cases where it would be desirable to have post-analysis DocValues based on multi-token fields.

The main use cases that I can think of are variants of faceting/terms aggregation. I understand that this could be viewed as "trappy" for the naive "Moby Dick word cloud" case; but:
1. I think this can be supported fairly cleanly in Lucene
1. Explicit user configuration of this option would help prevent people shooting themselves in the foot
1. The current situation is arguably "trappy" as well; it just offloads the trappiness onto Lucene-external workarounds for systems/users that want to support this kind of behavior
1. Integrating this functionality directly in Lucene would afford consistency guarantees that present opportunities for future optimizations (e.g., shared Terms dictionary between indexed terms and DocValues).

This issue proposes adding support for multi-token post-analysis DocValues directly to `IndexingChain`. The initial proposal involves extending the API to include `IndexableFieldType.tokenDocValuesType()` (in addition to existing `IndexableFieldType.docValuesType()`).

---
Migrated from [LUCENE-10023](https://issues.apache.org/jira/browse/LUCENE-10023) by Michael Gibney (@magibney), updated Oct 13 2021
Pull requests: https://github.com/apache/lucene/pull/208

Contributor guide

Open the contributing guide

Research direction

The issue names IndexingChain and the proposed IndexableFieldType.tokenDocValuesType() API; start by reading those APIs and the existing Analyzer.normalize and DocValues handling. Review the migrated LUCENE-10023 context and discussion to determine the design and scope. Done means an agreed implementation approach for multi-token post-analysis DocValues, with affected tests and files identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.