apache / apache/lucene

toString() methods on term/queries/etc are wrong: assume utf-8 encoded bytes. [LUCENE-2942]

Open
#4,016 3 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0-ALPHA legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In Lucene's trunk, a Term is just a Bytesref.
In a lot of cases this is a UTF-8 encoded string, but in some cases its not (e.g. collation fields).

The problem is that the toString methods all currently call utf8ToString().
This is wrong, though from a practical point of view i think just printing the bytes won't be very helpful for debugging most cases where the bytes really are utf-8 encoded.

So i think in these cases we should use the following technique: if the bytes are a valid utf-8 sequence, use BytesRef.utf8tostring(), otherwise just print the bytes: BytesRef.toString()

its no problem for performance because toString is only for debugging anyway.

---
Migrated from [LUCENE-2942](https://issues.apache.org/jira/browse/LUCENE-2942) by Robert Muir (@rmuir)
Attachments: [LUCENE-2942.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2942/LUCENE-2942.patch)

Contributor guide

Open the contributing guide

Research direction

Start by locating the Term, query, and related toString() methods that call BytesRef.utf8ToString(), then read BytesRef.utf8ToString() and BytesRef.toString(). Done means valid UTF-8 bytes retain readable output while invalid sequences use the byte representation, with the affected toString() methods covered by the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.