apache / apache/doris

[Bug] routine load 在非严格模式下导入 varchar 目标列时,非英文字符的长度截断不正确

Open
#64,334 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

doris-4.1.0-rc03-5960d4cea0e

### What's Wrong?

我在 Doris 中创建了如下的表
```sql
CREATE TABLE test_table
(
`env` VARCHAR(32) NOT NULL DEFAULT ''
)
ENGINE = OLAP
DUPLICATE KEY(env)
DISTRIBUTED BY RANDOM BUCKETS 1
properties(
"compression" = "zstd",
"enable_single_replica_compaction" = "true",
"replication_allocation" = "tag.location.default: 1",
"storage_format" = "V3"
);
```

同时开启了一个 routine load 任务
```sql
CREATE ROUTINE LOAD dwd.test_table ON test_table
COLUMNS(`env`)
PROPERTIES(
"format"="json",
"strict_mode"="false",
"max_filter_ratio" = "1",
"timezone" = "Asia/Shanghai",
"max_error_number" = "5000000",
"max_batch_interval" = "10",
"max_batch_rows" = "5000000",
"desired_concurrent_number" = "8",
"load_to_single_tablet" = "true",
"exec_mem_limit" = "8589934592"
)
FROM KAFKA(
"kafka_broker_list" = "127.0.0.1:9092",
"kafka_topic" = "test_load",
"property.kafka_default_offsets" = "OFFSET_END",
"property.group.id" = "test_load_doris"
);
```

之后,我向 kafka 中发送了一个消息:
```json
{"env":"${jnd${upper:ı}:ldap://test.comxxxxxx}"}
```

这个 json 中, `upper:` 字符后有一个特殊的 `U+0131` 字符;此时 Doris 的 routine load 任务会出现如下报错:
```
Reason: column_name[env], the length of input is too long than schema. first 32 bytes of input str: [${jnd${upper:ı}:ldap://test.com] schema length: 32; actual length: 33; . src line [];
```

同样的,导入如下的中文字符,也会出现问题:
```json
{"env":"中123456789012345678901234567890"}
```

报错如下:
```
Reason: column_name[env], the length of input is too long than schema. first 32 bytes of input str: [中12345678901234567890123456789] schema length: 32; actual length: 33; . src line [];
```

### What You Expected?

能够正确的按照 varchar 定义的长度截断字符串,并成功导入数据

### How to Reproduce?

_No response_

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.