apache / apache/lucene

Can we better take advantage of compact strings?

Open
#12,071 3 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

There's a non-negligible time that we spend on UTF-16 / UTF-8 conversions using our own `UnicodeUtil`, e.g. via the `BytesRef(String)` constructor. But since the introduction of compact strings, `String#getBytes(StandardCharsets.UTF_8)` has some fast tracks, e.g. if neither of the bytes has its highest bit set, then the string is an ASCII string that is the same as the UTF-8 representation.

I ran a quick microbenchmark that suggests that `String#getBytes` can indeed be significantly faster on ASCII strings:
- `charsetEncoder` leverages `StandardCharsets.UTF_8.encode(state.input)`
- `stringGetBytes` leverages `String#getBytes(StandardCharsets.UTF_8)`.
- `unicodeUtil` leverages `UnicodeUtil#UTF16toUTF8`.

```
Benchmark (input) Mode Cnt Score Error Units
ConversionBenchmark.charsetEncoder a thrpt 5 36.890 ± 0.572 ops/us
ConversionBenchmark.charsetEncoder abcdefghijklmnop thrpt 5 18.717 ± 1.367 ops/us
ConversionBenchmark.charsetEncoder recherché thrpt 5 10.098 ± 0.328 ops/us
ConversionBenchmark.stringGetBytes a thrpt 5 142.186 ± 18.849 ops/us
ConversionBenchmark.stringGetBytes abcdefghijklmnop thrpt 5 111.259 ± 2.203 ops/us
ConversionBenchmark.stringGetBytes recherché thrpt 5 53.565 ± 0.483 ops/us
ConversionBenchmark.unicodeUtil a thrpt 5 103.123 ± 3.970 ops/us
ConversionBenchmark.unicodeUtil abcdefghijklmnop thrpt 5 54.223 ± 1.342 ops/us
ConversionBenchmark.unicodeUtil recherché thrpt 5 58.166 ± 1.504 ops/us
```

Yet switching from `UnicodeUtil` to `String#getBytes` cannot be done transparently because they use a different replacement character for mismatched surrogate pairs. So I wonder if we have options for leveraging `String#getBytes` internally to make things a bit faster, of if this sort of things should be left for applications built on top of Lucene, e.g. using the `StringField(String, BytesRef, Store)` constructor instead of the `StringField(String, String, Store)` constructor and doing the UTF8 conversion themselves using `String#getBytes`.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing UnicodeUtil#UTF16toUTF8, the BytesRef(String) constructor, and the StringField(String, String, Store) path, then reproduce the reported ConversionBenchmark comparisons with String#getBytes(StandardCharsets.UTF_8). Pay particular attention to the differing replacement character for mismatched surrogate pairs; the issue does not name a target file or define a concrete acceptance criterion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.