pingcap / pingcap/tidb

expression: reduce UTF8 string builtin allocations by removing []rune conversion

Open
#66,398 0 comments 0 reactions 0 assignees View on GitHub
component/expression type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

## Enhancement

UTF8 string builtins in `pkg/expression` still use `[]rune(str)` in hot paths (both row-based and vectorized execution) for rune counting and slicing.

This causes avoidable allocations and extra CPU/GC overhead.

### Proposed improvement

Replace `[]rune`-based logic with zero-allocation UTF8 operations:

- `len([]rune(str))` -> `utf8.RuneCountInString(str)`
- `string([]rune(str)[:n])` / `string([]rune(str)[n:])` -> byte slicing at rune boundaries via a helper like `runeByteIndex(str, n)`
- Apply consistently in scalar + vectorized paths for:
- `LEFT`, `RIGHT`
- `LOCATE`
- `SUBSTR` / `MID` / `SUBSTRING`
- `INSERT`
- `LPAD`, `RPAD`
- `CHAR_LENGTH`

### Expected result

- Fewer allocations in expression evaluation
- Lower GC pressure
- No SQL behavior change for UTF8 semantics

### Validation plan

- Run targeted unit tests for affected UTF8 string builtins (scalar and vectorized)
- Add/compare benchmarks (allocs/op, ns/op) before and after
- Verify edge cases: multibyte characters, negative/out-of-range positions, empty substring/pad string, collation-sensitive locate behavior

Contributor guide

Open the contributing guide

Research direction

Start in pkg/expression by locating the []rune conversions used by the listed UTF8 string builtins, and compare their scalar and vectorized paths. Run targeted unit tests and inspect existing benchmarks before checking multibyte, boundary, padding, and collation-sensitive cases. Done means the affected paths avoid the stated allocations while preserving UTF8 SQL behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.