[expression] Vectorized LOCATE can silently delete accent-equivalent rows under the default collation
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
**Environment**
TiDB ed2376acc6/master 05b396fb66; vector ON; utf8mb4_0900_ai_ci; strict; MDL ON; one TiDB/PD/TiKV; no injection
**Steps**
Create a utf8mb4_0900_ai_ci table containing accented e, uppercase accented e, e, cafe with accented e, CAFE, x-accented-e-x, and xyz. Compare LOCATE(e,s) with tidb_enable_vectorized_expression OFF and ON. On two fresh copies run DELETE FROM t WHERE LOCATE(e,s)=0 under each mode, then compare affected rows and surviving primary keys. Keep MDL ON and use the default strict SQL mode.
### 2. What did you expect to see? (Required)
LOCATE follows its resolved collation in both scalar and vectorized execution. Only xyz is classified as not containing e, so the DELETE affects one row and preserves IDs 1 through 6.
### 3. What did you see instead? (Required)
Scalar execution deletes one row and preserves IDs 1,2,3,4,5,6. Default vectorized execution returns success, deletes five rows, and preserves only IDs 3 and 5. Four accent-equivalent business rows are silently lost. Binary-collation scalar and vector controls agree.
### 4. What is your TiDB version? (Required)
TiDB ed2376acc6/master 05b396fb66; vector ON; utf8mb4_0900_ai_ci; strict; MDL ON; one TiDB/PD/TiKV; no injection
Likely root cause and fix direction
**Likely root cause**
The scalar UTF8 LOCATE signatures call locateStringWithCollation, which searches collation keys. The vectorized signatures reduce IsCICollation to strings.ToLower followed by strings.Index. Case-insensitivity does not imply that lowercased code points implement accent, width, contraction, or expansion rules, so the fast evaluator silently changes predicate truth.
**Fix direction**
Make both vectorized UTF8 LOCATE signatures use the same collation-key search helper and position mapping as scalar execution. Add scalar/vector parity across accent-insensitive, accent-sensitive, binary, width, and expansion cases, including a complement DML consumer.
Contributor guide
Research direction
Start by reproducing the supplied LOCATE comparison with vectorization OFF and ON, including the complement DELETE. Read the scalar locateStringWithCollation path and the vectorized UTF8 LOCATE signatures mentioned in the report. Done means scalar and vectorized results agree across the listed collation cases and the DML preserves the same primary keys.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100