toString() methods on term/queries/etc are wrong: assume utf-8 encoded bytes. [LUCENE-2942]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
In Lucene's trunk, a Term is just a Bytesref.
In a lot of cases this is a UTF-8 encoded string, but in some cases its not (e.g. collation fields).
The problem is that the toString methods all currently call utf8ToString().
This is wrong, though from a practical point of view i think just printing the bytes won't be very helpful for debugging most cases where the bytes really are utf-8 encoded.
So i think in these cases we should use the following technique: if the bytes are a valid utf-8 sequence, use BytesRef.utf8tostring(), otherwise just print the bytes: BytesRef.toString()
its no problem for performance because toString is only for debugging anyway.
---
Migrated from [LUCENE-2942](https://issues.apache.org/jira/browse/LUCENE-2942) by Robert Muir (@rmuir)
Attachments: [LUCENE-2942.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2942/LUCENE-2942.patch)
Contributor guide
Research direction
Start by locating the Term, query, and related toString() methods that call BytesRef.utf8ToString(), then read BytesRef.utf8ToString() and BytesRef.toString(). Done means valid UTF-8 bytes retain readable output while invalid sequences use the byte representation, with the affected toString() methods covered by the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100