map unicode process-internal codepoints to replacement character [LUCENE-2019]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
A spinoff from #3091.
There are several process-internal codepoints in unicode, we should not store these in the index.
Instead they should be mapped to replacement character (U+FFFD), so they can be used process-internally.
An example of this is how Lucene Java currently uses U+FFFF process-internally, it can't be in the index or will cause problems.
---
Migrated from [LUCENE-2019](https://issues.apache.org/jira/browse/LUCENE-2019) by Robert Muir (@rmuir), updated Nov 30 2013
Attachments: [LUCENE-2019.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2019/LUCENE-2019.patch)
Contributor guide
Research direction
Start by reading the attached LUCENE-2019.patch and the surrounding issue discussion to identify the index-writing path affected by process-internal Unicode codepoints. Trace how U+FFFF is handled and determine where such codepoints should become U+FFFD. Done means process-internal codepoints are no longer stored in the index without breaking normal Unicode indexing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100