pingcap / pingcap/tidb

TiDB changes UNION result metadata when a REGEXP expression is read through a VIEW

Open
#70,641 2 comments 0 reactions 0 assignees View on GitHub
contribution severity/moderate sig/execution type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

TiDB resolves the result of a heterogeneous `UNION ALL` as binary data when
one branch is a `REGEXP` expression over `LONGTEXT` and another branch is an
ordinary `VARCHAR`. Reading the identical `REGEXP` expression through a
`VIEW` changes the final result to normal `utf8mb4` text.

The byte sequences are unchanged, but the MySQL protocol metadata is
observable to clients: PyMySQL returns `bytes` for the flat query and `str`
for the VIEW query. SQL-level `CHARSET()` and `COLLATION()` calls likewise
report `binary`/`binary` versus `utf8mb4`/`utf8mb4_bin`.

This is the stable mismatch from VECT round 5, pair 711, cut 2. The original
case reduced to two tables and one expression. It is distinct from the
previously reported TiDB `LONGTEXT` truncation bug: this case does not
truncate or otherwise change the payload.

### 1. Minimal reproduce step (Required)

Create a self-contained schema:

```sql
DROP DATABASE IF EXISTS tidb_union_regexp_view_repro;
CREATE DATABASE tidb_union_regexp_view_repro
CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
USE tidb_union_regexp_view_repro;

CREATE TABLE long_src (
s LONGTEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL
);

CREATE TABLE varchar_src (
s VARCHAR(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL
);

INSERT INTO long_src VALUES ('abc123');
INSERT INTO varchar_src VALUES ('sample_e');
```

Run the flat query:

```sql
SELECT s REGEXP '[a-zA-Z0-9]{3,12}' AS v FROM long_src
UNION ALL
SELECT s FROM varchar_src;
```

The displayed values are `1` and `sample_e`, but TiDB resolves the result as
binary. This SQL-only wrapper makes the type attributes visible:

```sql
SELECT v, CHARSET(v), COLLATION(v), HEX(v)
FROM (
SELECT s REGEXP '[a-zA-Z0-9]{3,12}' AS v FROM long_src
UNION ALL
SELECT s FROM varchar_src
) AS u;
```

Observed result:

```text
1 | binary | binary | 31
sample_e | binary | binary | 73616D706C655F65
```

Now store the same expression in a VIEW and repeat the UNION:

```sql
CREATE VIEW regexp_view AS
SELECT s REGEXP '[a-zA-Z0-9]{3,12}' AS v FROM long_src;

SELECT v, CHARSET(v), COLLATION(v), HEX(v)
FROM (
SELECT v FROM regexp_view
UNION ALL
SELECT s FROM varchar_src
) AS u;
```

Observed result:

```text
1 | utf8mb4 | utf8mb4_bin | 31
sample_e | utf8mb4 | utf8mb4_bin | 73616D706C655F65
```

`SHOW COLUMNS FROM regexp_view` reports the VIEW column as `BIGINT`. The
direct expression also has protocol type `MYSQL_TYPE_LONGLONG` when selected
by itself, so crossing the VIEW boundary should not introduce a different
logical input type for UNION resolution.

## Protocol and plan evidence

Using PyMySQL 1.1.2, the final column metadata and decoded values are:

```text
flat: type=MYSQL_TYPE_VAR_STRING, charsetnr=63, length=40
values=(b'1', b'sample_e')

VIEW: type=MYSQL_TYPE_VAR_STRING, charsetnr=46, length=80
values=('1', 'sample_e')
```

Charset number 63 is the MySQL protocol's `binary` character set. Charset
number 46 is `utf8mb4_bin`; its name contains `_bin`, but it remains an
`utf8mb4` text collation and clients decode it as text.

`EXPLAIN` shows the inconsistent UNION coercions directly:

```text
flat REGEXP branch: cast(regexp(...), varbinary(40) BINARY)
flat VARCHAR branch: cast(varchar_src.s, varbinary(40) BINARY)

VIEW REGEXP branch: cast(regexp(...), varchar(20) CHARACTER SET utf8mb4
COLLATE utf8mb4_bin)
VIEW VARCHAR branch: cast(varchar_src.s, varchar(20) CHARACTER SET utf8mb4
COLLATE utf8mb4_bin)
```

A derived table or CTE around the expression still produces binary metadata.
An explicit `CAST(... AS SIGNED)` produces the same `utf8mb4` result as the
VIEW. This narrows the issue to type aggregation for the raw `REGEXP` scalar
expression versus the `BIGINT` column type recorded for the VIEW.

### 2. What did you expect to see? (Required)

The flat expression and its VIEW column both have the logical type `BIGINT`.
Combining either one with the same `VARCHAR ... CHARACTER SET utf8mb4` branch
should therefore produce the same text character set and protocol metadata.
In particular, merely exposing the expression through a VIEW must not change
whether clients receive text or binary data.

### 3. What did you see instead (Required)

### 4. What is your TiDB version? (Required)

```text
TiDB version (SELECT tidb_version()):
Release Version: v8.5.7
Edition: Community
Git Commit Hash: 202b7f47286a1109b5c957401d34c9358d130ae0
Git Branch: HEAD
UTC Build Time: 2026-07-15 02:06:00
GoVersion: go1.25.10
Race Enabled: false
Check Table Before Drop: false
Store: unistore
```

Contributor guide

Open the contributing guide

Research direction

Start by running the self-contained SQL reproduction and comparing the flat UNION with the VIEW query, including EXPLAIN and protocol metadata. Trace UNION type aggregation for the raw REGEXP scalar expression versus the VIEW's BIGINT column, then verify that both paths produce matching utf8mb4 metadata without changing the payload.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mysql, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.