Multi-token post-analysis DocValues [LUCENE-10023]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
The single-token case for post-analysis DocValues is accounted for by `Analyzer.normalize(...)` (and formerly `MultiTermAwareComponent`); but there are cases where it would be desirable to have post-analysis DocValues based on multi-token fields.
The main use cases that I can think of are variants of faceting/terms aggregation. I understand that this could be viewed as "trappy" for the naive "Moby Dick word cloud" case; but:
1. I think this can be supported fairly cleanly in Lucene
1. Explicit user configuration of this option would help prevent people shooting themselves in the foot
1. The current situation is arguably "trappy" as well; it just offloads the trappiness onto Lucene-external workarounds for systems/users that want to support this kind of behavior
1. Integrating this functionality directly in Lucene would afford consistency guarantees that present opportunities for future optimizations (e.g., shared Terms dictionary between indexed terms and DocValues).
This issue proposes adding support for multi-token post-analysis DocValues directly to `IndexingChain`. The initial proposal involves extending the API to include `IndexableFieldType.tokenDocValuesType()` (in addition to existing `IndexableFieldType.docValuesType()`).
---
Migrated from [LUCENE-10023](https://issues.apache.org/jira/browse/LUCENE-10023) by Michael Gibney (@magibney), updated Oct 13 2021
Pull requests: https://github.com/apache/lucene/pull/208
Contributor guide
Research direction
The issue names IndexingChain and the proposed IndexableFieldType.tokenDocValuesType() API; start by reading those APIs and the existing Analyzer.normalize and DocValues handling. Review the migrated LUCENE-10023 context and discussion to determine the design and scope. Done means an agreed implementation approach for multi-token post-analysis DocValues, with affected tests and files identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100