matrixorigin / matrixorigin/matrixone
[Bug]: binary string functions use character semantics instead of byte semantics
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Bug Report
### Description
MatrixOne evaluates several string functions over `_binary` / `BLOB` / `VARBINARY` values using UTF-8 character semantics, while MySQL treats binary strings as byte sequences.
This affects at least:
- `REVERSE`
- `SUBSTRING`
- `LEFT`
- `RIGHT`
- `LOCATE`
- `INSTR`
- `LOWER`
- `UPPER`
`CHAR_LENGTH(_binary ...)` is already tracked separately in #25295, and binary charset/collation metadata is tracked in #25300. This issue tracks function result semantics beyond metadata and length.
### Steps to Reproduce
```sql
select
hex(_binary '你好') as bin_utf8_hex,
length(_binary '你好') as bin_len,
char_length(_binary '你好') as bin_char_len;
select
hex(reverse('你好')) as reverse_text_hex,
hex(reverse(_binary '你好')) as reverse_binary_hex,
hex(reverse(cast('你好' as binary))) as reverse_cast_binary_hex;
select
hex(substring('你好', 2, 1)) as substr_text_hex,
hex(substring(_binary '你好', 2, 1)) as substr_bin_hex,
hex(substring(_binary '你好', 2, 2)) as substr_bin_2_hex,
hex(left(_binary '你好', 2)) as left_bin_2_hex,
hex(right(_binary '你好', 2)) as right_bin_2_hex;
select
locate(_binary X'A0', _binary '你好') as locate_bin_byte,
instr(_binary '你好', _binary X'A0') as instr_bin_byte,
locate('好', _binary '你好') as locate_text_in_bin,
locate(_binary '好', '你好') as locate_bin_in_text;
select
hex(lower(_binary 'ABC')) as lower_bin_hex,
hex(upper(_binary 'abc')) as upper_bin_hex,
hex(lower('ABC')) as lower_text_hex,
hex(upper('abc')) as upper_text_hex;
create table t_blob_string_funcs (
id int primary key,
b blob,
vb varbinary(20)
);
insert into t_blob_string_funcs values
(1, _binary '你好', _binary '你好'),
(2, X'20E4BDA020', X'20E4BDA020');
select id,
length(b) as len_b,
char_length(b) as char_len_b,
hex(reverse(b)) as rev_b_hex,
hex(substring(b, 2, 2)) as substr_b_hex,
hex(trim(b)) as trim_b_hex
from t_blob_string_funcs
order by id;
select id,
length(vb) as len_vb,
char_length(vb) as char_len_vb,
hex(reverse(vb)) as rev_vb_hex,
hex(substring(vb, 2, 2)) as substr_vb_hex,
hex(trim(vb)) as trim_vb_hex
from t_blob_string_funcs
order by id;
```
### MySQL Result
Key rows:
```text
bin_utf8_hex bin_len bin_char_len
E4BDA0E5A5BD 6 6
reverse_text_hex reverse_binary_hex reverse_cast_binary_hex
E5A5BDE4BDA0 BDA5E5A0BDE4 BDA5E5A0BDE4
substr_text_hex substr_bin_hex substr_bin_2_hex left_bin_2_hex right_bin_2_hex
E5A5BD BD BDA0 E4BD A5BD
locate_bin_byte instr_bin_byte locate_text_in_bin locate_bin_in_text
3 3 4 2
lower_bin_hex upper_bin_hex lower_text_hex upper_text_hex
414243 616263 616263 414243
```
For `BLOB` / `VARBINARY`, MySQL also applies byte semantics:
```text
id len_b char_len_b rev_b_hex substr_b_hex trim_b_hex
1 6 6 BDA5E5A0BDE4 BDA0 E4BDA0E5A5BD
2 5 5 20A0BDE420 E4BD E4BDA0
id len_vb char_len_vb rev_vb_hex substr_vb_hex trim_vb_hex
1 6 6 BDA5E5A0BDE4 BDA0 E4BDA0E5A5BD
2 5 5 20A0BDE420 E4BD E4BDA0
```
### MatrixOne Result
Tested on latest `main` at `200bfa7e0efb`.
Key rows:
```text
bin_utf8_hex bin_len bin_char_len
E4BDA0E5A5BD 6 2
reverse_text_hex reverse_binary_hex reverse_cast_binary_hex
E5A5BDE4BDA0 E5A5BDE4BDA0 E5A5BDE4BDA0
substr_text_hex substr_bin_hex substr_bin_2_hex left_bin_2_hex right_bin_2_hex
E5A5BD E5A5BD E5A5BD E4BDA0E5A5BD E4BDA0E5A5BD
locate_bin_byte instr_bin_byte locate_text_in_bin locate_bin_in_text
0 0 2 2
lower_bin_hex upper_bin_hex lower_text_hex upper_text_hex
616263 414243 616263 414243
```
For `BLOB` / `VARBINARY`, MatrixOne also uses character semantics:
```text
id len_b char_len_b rev_b_hex substr_b_hex trim_b_hex
1 6 2 E5A5BDE4BDA0 E5A5BD E4BDA0E5A5BD
2 5 3 20E4BDA020 E4BDA020 E4BDA0
id len_vb char_len_vb rev_vb_hex substr_vb_hex trim_vb_hex
1 6 2 E5A5BDE4BDA0 E5A5BD E4BDA0E5A5BD
2 5 3 20E4BDA020 E4BDA020 E4BDA0
```
### Expected Behavior
Binary strings should be treated as byte sequences for these string functions, matching MySQL.
### Notes
Found during active MySQL compatibility exploration, Module 4: String / Charset / Collation. I searched existing issues for binary string byte semantics in `REVERSE` / `SUBSTRING` / `LOCATE` / `LOWER` / `UPPER` and did not find a duplicate.
Contributor guide
Assessment
This issue has not been assessed yet.