apache / apache/lucene

CJKBigramFilter produces inconsistent token positions with outputUnigrams enabled vs disabled

Open
#15,812 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

`CJKBigramFilter` produces different token positions for the same input depending on whether `outputUnigrams` is `true` or `false`. This causes phrase query mismatches when index-time and search-time analyzers use different `outputUnigrams` settings.

### Steps to reproduce

Using the `_analyze` API (tested on ES 9.3.1 / Lucene 10.2.1):

**With `outputUnigrams: true`:**
```json
POST /_analyze
{
"tokenizer": "standard",
"filter": [{ "type": "cjk_bigram", "output_unigrams": true }],
"text": "一二、三"
}
```

Result — `三` is at **position 2**:
```
一 position: 0
一二 position: 0 (positionLength: 2)
二 position: 1
三 position: 2
```

**With `outputUnigrams: false`:**
```json
POST /_analyze
{
"tokenizer": "standard",
"filter": [{ "type": "cjk_bigram", "output_unigrams": false }],
"text": "一二、三"
}
```

Result — `三` is at **position 1**:
```
一二 position: 0
三 position: 1
```

### Expected behavior

`三` should be at the same effective position regardless of the `outputUnigrams` setting. With `outputUnigrams: false`, `三` should be at position 2 (or equivalently, the bigram `一二` should account for occupying two character positions).

### Analysis

The issue is in `flushBigram()`. When `outputUnigrams=false`, bigrams are emitted with the default `positionIncrement=1` (from `clearAttributes()`), but a bigram conceptually spans two character positions. After a word break (e.g. punctuation `、`), a subsequent lone CJK character gets a position that differs from the `outputUnigrams=true` case because the bigram only advanced the position counter by 1 instead of 2.

This specifically manifests when:
1. A CJK segment is followed by a word break (punctuation, whitespace, non-CJK text)
2. The preceding CJK segment has an even number of characters (so it is fully consumed by bigrams with no trailing unigram)

### Impact

This breaks phrase search when using a combined unigram+bigram indexing strategy with bigram-only search queries, which is a common optimization pattern for CJK search. The workaround is to enable `outputUnigrams` on both index and search sides, at the cost of generating redundant unigrams at search time.

### Version

Confirmed on Lucene 10.2.1 (Elasticsearch 9.3.1). Also present on Lucene `main` branch as of 2026-03-10.

Contributor guide

Open the contributing guide

Research direction

Start with CJKBigramFilter.flushBigram() and reproduce the discrepancy through the _analyze API using the two outputUnigrams settings and the input "一二、三". Trace how position increments are assigned after an even-length CJK segment and word break; done means the subsequent character has the same effective position in both configurations and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.