apache / apache/doris

[Bug] FE reports hardcoded field length 255 for all VARCHAR columns in the MySQL protocol column definition packet

Open
#67,349 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

Doris version doris-3.1.1-rc01-9378ac80
Client side:
MySQL Connector/ODBC 8.1 Unicode Driver (Windows)
mysql CLI (Linux)

### What's Wrong?

When a result set is returned over the MySQL wire protocol, the FE puts a constant 255 into the length field of every column definition packet for VARCHAR columns, regardless of the length declared in the DDL.

The actual row data is transmitted correctly and in full — only the metadata is wrong. The mismatch is visible directly in the --column-type-info output, where Length stays at 255 while Max_length correctly reports 900:

Field 3: `c_1000`
Type: STRING
Collation: utf8mb3_general_ci (33)
Length: 255 <-- declared VARCHAR(1000)
Max_length: 900

Field 7: `c_65533`
Type: STRING
Collation: utf8mb3_general_ci (33)
Length: 255 <-- declared VARCHAR(65533)
Max_length: 900

There is also an internal inconsistency inside a single ODBC session: SQLColumns (catalog metadata) returns the correct declared lengths, while SQLDescribeCol (result set metadata) returns a value derived from the hardcoded 255.

Practical impact

Client applications that size their buffers from result set metadata silently truncate string values. With a Unicode ODBC driver the reported 255 bytes are divided by the wide-character size, producing ColumnSize = 127, so values are cut at roughly 255–256 bytes:

Doris reports length = 255
-> Connector/ODBC 8.1(w) reports ColumnSize = 127
-> client allocates a 127-character buffer
-> string values are truncated

This was originally observed as silent truncation of long text fields in a BI tool loading data through the MySQL ODBC driver. No error or warning is raised on either side — the data simply arrives incomplete.

There is no server-side workaround. CAST(col AS VARCHAR(4000)) does not change the reported length, and narrowing the column with ALTER TABLE ... MODIFY COLUMN has no effect either, since 255 is reported unconditionally.

### What You Expected?

The length field of the column definition packet should reflect the declared column length (in bytes, consistent with Doris VARCHAR(M) semantics), the same way MySQL does. SQLDescribeCol and SQLColumns should agree with each other.

### How to Reproduce?

1. Create the table and insert data

sql
CREATE DATABASE IF NOT EXISTS odbc_test;

USE odbc_test;

CREATE TABLE t_types (
id INT NOT NULL,
c_255 VARCHAR(255) NULL,
c_1000 VARCHAR(1000) NULL,
c_4000 VARCHAR(4000) NULL,
c_8000 VARCHAR(8000) NULL,
c_16384 VARCHAR(16384) NULL,
c_65533 VARCHAR(65533) NULL
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num" = "3");

INSERT INTO t_types VALUES
(1, REPEAT('A',255), REPEAT('B',900), REPEAT('C',900),
REPEAT('D',900), REPEAT('E',900), REPEAT('F',900));

2. Inspect the protocol metadata

bash
mysql --column-type-info -h <FE_HOST> -P 9030 -u <USER> -p \

-e "SELECT * FROM odbc_test.t_types LIMIT 1"

Observed: Length: 255 for every string column, including c_65533.
Expected: 1000, 4000, 8000, 16384 and 65533 respectively.

3. Confirm the inconsistency through ODBC (optional)

powershell
$c = New-Object System.Data.Odbc.OdbcConnection("DSN=<DSN>")

$c.Open()

# Result set metadata -> ColumnSize 127 for every string column
$cmd = $c.CreateCommand()
$cmd.CommandText = "SELECT * FROM odbc_test.t_types LIMIT 1"
$r = $cmd.ExecuteReader()
$schema = $r.GetSchemaTable()
$r.Close()
$schema | Select-Object ColumnName, ColumnSize, ProviderType | Format-Table

# Catalog metadata -> correct declared lengths
$c.GetSchema("Columns", @("odbc_test", $null, "t_types", $null)) |
Select-Object COLUMN_NAME, TYPE_NAME, COLUMN_SIZE | Format-Table
$c.Close()

Observed GetSchemaTable() output:

ColumnName  ColumnSize  ProviderType

---------- ---------- ------------
id 4 10
c_255 127 11
c_1000 127 11
c_4000 127 11
c_8000 127 11
c_16384 127 11
c_65533 127 11

Observed GetSchema("Columns") output (correct):

COLUMN_NAME  TYPE_NAME  COLUMN_SIZE

----------- --------- -----------
c_255 varchar 255
c_1000 varchar 1000
c_4000 varchar 4000
c_8000 varchar 8000
c_16384 varchar 16384
c_65533 varchar 65533

Reading the values themselves through the same connection returns all 900
characters, confirming that only the metadata is affected.

4. Control experiment against MySQL 8.0

To rule out the client and the driver, the same test was run against MySQL 8.0
(official mysql:8.0 Docker image) using the same mysql CLI and the same
command. The table was created with DEFAULT CHARSET=latin1 so that the
declared lengths are directly comparable in bytes, and the 65 KB column was put
in a separate table to stay within the MySQL row size limit.

sql
CREATE TABLE t_types (

id INT NOT NULL,
c_255 VARCHAR(255), c_1000 VARCHAR(1000), c_4000 VARCHAR(4000),
c_8000 VARCHAR(8000), c_16384 VARCHAR(16384)
) DEFAULT CHARSET=latin1;

CREATE TABLE t_big (
id INT NOT NULL,
c_65000 VARCHAR(65000)
) DEFAULT CHARSET=latin1;

MySQL 8.0 reports the declared length of each column individually:


Column | Declared | MySQL 8.0 Length | Doris 3.1.1 Length
-- | -- | -- | --
c_255 | VARCHAR(255) | 255 | 255
c_1000 | VARCHAR(1000) | 1000 | 255
c_4000 | VARCHAR(4000) | 4000 | 255
c_8000 | VARCHAR(8000) | 8000 | 255
c_16384 | VARCHAR(16384) | 16384 | 255
65 KB col | see above | 65000 | 255

Max_length is 900 on both servers, confirming that the row data itself is
transmitted correctly in either case.

Two further differences appear in the same packets:


  • MySQL reports the actual collation of the column (latin1_swedish_ci (8)),
    while Doris always reports utf8mb3_general_ci (33).

  • MySQL reports the column type as VAR_STRING, Doris reports STRING.

### Anything Else?

The value appears to originate in [https://github.com/apache/doris/blob/ded08aebefdb76b167c1f5fa164feaa1b4732205/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlSerializer.java#L271](url).

In getMysqlTypeLength(Type type), numeric and temporal types are handled explicitly while CHAR/VARCHAR fall through to the default branch, which carries an existing todo acknowledging that the declared field length is not used yet:

java
// todo:It needs to be obtained according to the field length set during the actual creation,
// todo:which is not supported for the time being.default is 255
// CHAR,VARCHAR:
default:
return 255;

The result is written into the column definition packet via writeInt4(getMysqlTypeLength(...)) in the three writeField(...) overloads. The collation in the same packet is likewise a constant, writeInt2(33), which matches the observed utf8mb3_general_ci (33).

Additional notes:

The server reports utf8mb3_general_ci (33) as the collation even when the client connects with charset=utf8mb4. This may be unrelated, but it affects how drivers convert the reported byte length into a character count.
The server version string is reported as 5.7.99, so clients apply MySQL 5.7 compatibility behaviour when interpreting these metadata

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start at the FE MySQL wire-protocol handling for column definition packets and reproduce the issue with the provided CREATE TABLE, INSERT, and mysql --column-type-info commands. Trace how VARCHAR metadata length is populated, then verify that declared lengths appear in protocol output and that SQLDescribeCol agrees with SQLColumns without truncating returned values.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.