apache / apache/doris-flink-connector
[Enhancement] Adjust VARCHAR byte length calculation to fully support UTF-8 (utf8mb4)
- Dominant language
- Java
- Stars
- 385
- Forks
- 279
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 6
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
The current logic in `TypeConverter.java` uses a multiplier of `3` to calculate the required byte length for the Doris `VARCHAR` type:
```java
// Current implementation
return length * 3 > 65533
? DorisType.STRING
: String.format("%s(%s)", DorisType.VARCHAR, length * 3);
```
This assumes a maximum of 3 bytes per character, which is insufficient for the widely used utf8mb4 character set (common in MySQL/MariaDB and other sources). The utf8mb4 encoding supports the full range of Unicode characters (including emojis), requiring up to 4 bytes per character.
If a source column contains 4-byte characters, the calculated byte length may underestimate the required size, leading to:
Data truncation or corruption during the synchronization process.
Load failures with errors such as "data length exceeded" or "row size too large" when Doris enforces the byte limit.
### Solution
This change updates the byte multiplier from 3 to 4 to safely accommodate the full utf8mb4 character set, ensuring the calculated byte length is always sufficient for the defined character length, thus guaranteeing data integrity and preventing sync failures.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start in TypeConverter.java at the VARCHAR byte-length calculation and review how the current multiplier and 65533 limit are applied. Update the calculation to account for up to four UTF-8 bytes per character, then verify that long utf8mb4 values do not produce an undersized VARCHAR definition or exceed the Doris limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100