matrixorigin / matrixorigin/matrixone

[Bug]: binary string functions use character semantics instead of byte semantics

Open
#25,301 2 comments 0 reactions 1 assignee Claimed by @ck89119 View on GitHub
kind/bug severity/s1
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Bug Report

### Description

MatrixOne evaluates several string functions over `_binary` / `BLOB` / `VARBINARY` values using UTF-8 character semantics, while MySQL treats binary strings as byte sequences.

This affects at least:

- `REVERSE`
- `SUBSTRING`
- `LEFT`
- `RIGHT`
- `LOCATE`
- `INSTR`
- `LOWER`
- `UPPER`

`CHAR_LENGTH(_binary ...)` is already tracked separately in #25295, and binary charset/collation metadata is tracked in #25300. This issue tracks function result semantics beyond metadata and length.

### Steps to Reproduce

```sql
select
hex(_binary '你好') as bin_utf8_hex,
length(_binary '你好') as bin_len,
char_length(_binary '你好') as bin_char_len;

select
hex(reverse('你好')) as reverse_text_hex,
hex(reverse(_binary '你好')) as reverse_binary_hex,
hex(reverse(cast('你好' as binary))) as reverse_cast_binary_hex;

select
hex(substring('你好', 2, 1)) as substr_text_hex,
hex(substring(_binary '你好', 2, 1)) as substr_bin_hex,
hex(substring(_binary '你好', 2, 2)) as substr_bin_2_hex,
hex(left(_binary '你好', 2)) as left_bin_2_hex,
hex(right(_binary '你好', 2)) as right_bin_2_hex;

select
locate(_binary X'A0', _binary '你好') as locate_bin_byte,
instr(_binary '你好', _binary X'A0') as instr_bin_byte,
locate('好', _binary '你好') as locate_text_in_bin,
locate(_binary '好', '你好') as locate_bin_in_text;

select
hex(lower(_binary 'ABC')) as lower_bin_hex,
hex(upper(_binary 'abc')) as upper_bin_hex,
hex(lower('ABC')) as lower_text_hex,
hex(upper('abc')) as upper_text_hex;

create table t_blob_string_funcs (
id int primary key,
b blob,
vb varbinary(20)
);

insert into t_blob_string_funcs values
(1, _binary '你好', _binary '你好'),
(2, X'20E4BDA020', X'20E4BDA020');

select id,
length(b) as len_b,
char_length(b) as char_len_b,
hex(reverse(b)) as rev_b_hex,
hex(substring(b, 2, 2)) as substr_b_hex,
hex(trim(b)) as trim_b_hex
from t_blob_string_funcs
order by id;

select id,
length(vb) as len_vb,
char_length(vb) as char_len_vb,
hex(reverse(vb)) as rev_vb_hex,
hex(substring(vb, 2, 2)) as substr_vb_hex,
hex(trim(vb)) as trim_vb_hex
from t_blob_string_funcs
order by id;
```

### MySQL Result

Key rows:

```text
bin_utf8_hex bin_len bin_char_len
E4BDA0E5A5BD 6 6

reverse_text_hex reverse_binary_hex reverse_cast_binary_hex
E5A5BDE4BDA0 BDA5E5A0BDE4 BDA5E5A0BDE4

substr_text_hex substr_bin_hex substr_bin_2_hex left_bin_2_hex right_bin_2_hex
E5A5BD BD BDA0 E4BD A5BD

locate_bin_byte instr_bin_byte locate_text_in_bin locate_bin_in_text
3 3 4 2

lower_bin_hex upper_bin_hex lower_text_hex upper_text_hex
414243 616263 616263 414243
```

For `BLOB` / `VARBINARY`, MySQL also applies byte semantics:

```text
id len_b char_len_b rev_b_hex substr_b_hex trim_b_hex
1 6 6 BDA5E5A0BDE4 BDA0 E4BDA0E5A5BD
2 5 5 20A0BDE420 E4BD E4BDA0

id len_vb char_len_vb rev_vb_hex substr_vb_hex trim_vb_hex
1 6 6 BDA5E5A0BDE4 BDA0 E4BDA0E5A5BD
2 5 5 20A0BDE420 E4BD E4BDA0
```

### MatrixOne Result

Tested on latest `main` at `200bfa7e0efb`.

Key rows:

```text
bin_utf8_hex bin_len bin_char_len
E4BDA0E5A5BD 6 2

reverse_text_hex reverse_binary_hex reverse_cast_binary_hex
E5A5BDE4BDA0 E5A5BDE4BDA0 E5A5BDE4BDA0

substr_text_hex substr_bin_hex substr_bin_2_hex left_bin_2_hex right_bin_2_hex
E5A5BD E5A5BD E5A5BD E4BDA0E5A5BD E4BDA0E5A5BD

locate_bin_byte instr_bin_byte locate_text_in_bin locate_bin_in_text
0 0 2 2

lower_bin_hex upper_bin_hex lower_text_hex upper_text_hex
616263 414243 616263 414243
```

For `BLOB` / `VARBINARY`, MatrixOne also uses character semantics:

```text
id len_b char_len_b rev_b_hex substr_b_hex trim_b_hex
1 6 2 E5A5BDE4BDA0 E5A5BD E4BDA0E5A5BD
2 5 3 20E4BDA020 E4BDA020 E4BDA0

id len_vb char_len_vb rev_vb_hex substr_vb_hex trim_vb_hex
1 6 2 E5A5BDE4BDA0 E5A5BD E4BDA0E5A5BD
2 5 3 20E4BDA020 E4BDA020 E4BDA0
```

### Expected Behavior

Binary strings should be treated as byte sequences for these string functions, matching MySQL.

### Notes

Found during active MySQL compatibility exploration, Module 4: String / Charset / Collation. I searched existing issues for binary string byte semantics in `REVERSE` / `SUBSTRING` / `LOCATE` / `LOWER` / `UPPER` and did not find a duplicate.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.