Can we better take advantage of compact strings?
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
There's a non-negligible time that we spend on UTF-16 / UTF-8 conversions using our own `UnicodeUtil`, e.g. via the `BytesRef(String)` constructor. But since the introduction of compact strings, `String#getBytes(StandardCharsets.UTF_8)` has some fast tracks, e.g. if neither of the bytes has its highest bit set, then the string is an ASCII string that is the same as the UTF-8 representation.
I ran a quick microbenchmark that suggests that `String#getBytes` can indeed be significantly faster on ASCII strings:
- `charsetEncoder` leverages `StandardCharsets.UTF_8.encode(state.input)`
- `stringGetBytes` leverages `String#getBytes(StandardCharsets.UTF_8)`.
- `unicodeUtil` leverages `UnicodeUtil#UTF16toUTF8`.
```
Benchmark (input) Mode Cnt Score Error Units
ConversionBenchmark.charsetEncoder a thrpt 5 36.890 ± 0.572 ops/us
ConversionBenchmark.charsetEncoder abcdefghijklmnop thrpt 5 18.717 ± 1.367 ops/us
ConversionBenchmark.charsetEncoder recherché thrpt 5 10.098 ± 0.328 ops/us
ConversionBenchmark.stringGetBytes a thrpt 5 142.186 ± 18.849 ops/us
ConversionBenchmark.stringGetBytes abcdefghijklmnop thrpt 5 111.259 ± 2.203 ops/us
ConversionBenchmark.stringGetBytes recherché thrpt 5 53.565 ± 0.483 ops/us
ConversionBenchmark.unicodeUtil a thrpt 5 103.123 ± 3.970 ops/us
ConversionBenchmark.unicodeUtil abcdefghijklmnop thrpt 5 54.223 ± 1.342 ops/us
ConversionBenchmark.unicodeUtil recherché thrpt 5 58.166 ± 1.504 ops/us
```
Yet switching from `UnicodeUtil` to `String#getBytes` cannot be done transparently because they use a different replacement character for mismatched surrogate pairs. So I wonder if we have options for leveraging `String#getBytes` internally to make things a bit faster, of if this sort of things should be left for applications built on top of Lucene, e.g. using the `StringField(String, BytesRef, Store)` constructor instead of the `StringField(String, String, Store)` constructor and doing the UTF8 conversion themselves using `String#getBytes`.
Contributor guide
Research direction
Start by reviewing UnicodeUtil#UTF16toUTF8, the BytesRef(String) constructor, and the StringField(String, String, Store) path, then reproduce the reported ConversionBenchmark comparisons with String#getBytes(StandardCharsets.UTF_8). Pay particular attention to the differing replacement character for mismatched surrogate pairs; the issue does not name a target file or define a concrete acceptance criterion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100