apache / apache/lucene

map unicode process-internal codepoints to replacement character [LUCENE-2019]

Open
#3,094 30 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:core/index type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

A spinoff from #3091.

There are several process-internal codepoints in unicode, we should not store these in the index.
Instead they should be mapped to replacement character (U+FFFD), so they can be used process-internally.

An example of this is how Lucene Java currently uses U+FFFF process-internally, it can't be in the index or will cause problems.

---
Migrated from [LUCENE-2019](https://issues.apache.org/jira/browse/LUCENE-2019) by Robert Muir (@rmuir), updated Nov 30 2013
Attachments: [LUCENE-2019.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2019/LUCENE-2019.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reading the attached LUCENE-2019.patch and the surrounding issue discussion to identify the index-writing path affected by process-internal Unicode codepoints. Trace how U+FFFF is handled and determine where such codepoints should become U+FFFD. Done means process-internal codepoints are no longer stored in the index without breaking normal Unicode indexing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.