pingcap / pingcap/tidb

Dumpling: support specifying the character set of the output files

Open
#54,217 1 comment 0 reactions 0 assignees View on GitHub
affects-6.5 component/dumpling report/customer type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**

Currently dumpling assumes the output is always utf8mb4. If a user wants the result in a different character set such as GBK, they will either need `iconv` or changing `@@character_set_results`. The former requires 2x the storage space while the latter may cause unexpected transformation like insertion of unwanted `\` and `"`.

**Describe the feature you'd like:**

Add a flag that specifies the output encoding (naming and options follow that of Lightning):

* `--data-character-set=«encoding»`, should support:
* utf8mb4
* gb18030
* (optional) gbk
* (optional) latin1
* (optional) `--schema-character-set=«encoding»`, default to `--data-character-set` if unspecified
* (optional) `--data-invalid-char-replace='?'`, what to do with characters outside of the specified charset.

Both CSV and SQL output format should be supported, and should round-trip with the corresponding Lightning settings (including the cases with custom CSV separator/terminator etc)

**Describe alternatives you've considered:**

Nothing, document that Dumpling can only output utf8mb4 and require users to use `iconv` to perform conversion.

**Teachability, Documentation, Adoption, Migration Strategy:**

Contributor guide

Open the contributing guide

Research direction

Start by locating Dumpling's flag definitions and its CSV and SQL output paths, then compare the corresponding character-set options in Lightning. Implement the requested data and schema encoding behavior, including invalid-character handling if supported, and verify round-tripping with matching Lightning settings and custom CSV separators or terminators.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mysql
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.