unexpected error "encoding failed" when grouping by a column cast to binary

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
```sql
DROP TABLE IF EXISTS tb0;
CREATE TABLE tb0 (
c2 VARBINARY(102) NOT NULL DEFAULT x''
);

INSERT INTO tb0 (c2) VALUES (x'E32B17FD3503E33826C90B1D');
SELECT CAST(t0.c2 AS BINARY) AS col_0 FROM tb0 AS t0 group by col_0;
```

### 2. What did you expect to see? (Required)
MySQL behavior:
```sql
mysql> SELECT CAST(t0.c2 AS BINARY) AS col_0 FROM tb0 AS t0 group by col_0;
+----------------------------+
| col_0 |
+----------------------------+
| 0xE32B17FD3503E33826C90B1D |
+----------------------------+
1 row in set (0.001 sec)
```
### 3. What did you see instead (Required)
```sql
mysql> SELECT CAST(t0.c2 AS BINARY) AS col_0 FROM tb0 AS t0 group by col_0;
ERROR 1105 (HY000): encoding failed
```
### 4. What is your TiDB version? (Required)
```
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version() |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v9.0.0-beta.2.pre-250-ga6d415f24a
Edition: Community
Git Commit Hash: a6d415f24a7d182ebca3d189c457d9044537028f
Git Branch: HEAD
UTC Build Time: 2025-08-03 10:01:12
GoVersion: go1.23.11
Race Enabled: false
Check Table Before Drop: false
Store: tikv
Kernel Type: Classic |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.001 sec)
```