Dumpling: support specifying the character set of the output files
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
Currently dumpling assumes the output is always utf8mb4. If a user wants the result in a different character set such as GBK, they will either need `iconv` or changing `@@character_set_results`. The former requires 2x the storage space while the latter may cause unexpected transformation like insertion of unwanted `\` and `"`.
**Describe the feature you'd like:**
Add a flag that specifies the output encoding (naming and options follow that of Lightning):
* `--data-character-set=«encoding»`, should support:
* utf8mb4
* gb18030
* (optional) gbk
* (optional) latin1
* (optional) `--schema-character-set=«encoding»`, default to `--data-character-set` if unspecified
* (optional) `--data-invalid-char-replace='?'`, what to do with characters outside of the specified charset.
Both CSV and SQL output format should be supported, and should round-trip with the corresponding Lightning settings (including the cases with custom CSV separator/terminator etc)
**Describe alternatives you've considered:**
Nothing, document that Dumpling can only output utf8mb4 and require users to use `iconv` to perform conversion.
**Teachability, Documentation, Adoption, Migration Strategy:**
Contributor guide
Research direction
Start by locating Dumpling's flag definitions and its CSV and SQL output paths, then compare the corresponding character-set options in Lightning. Implement the requested data and schema encoding behavior, including invalid-character handling if supported, and verify round-tripping with matching Lightning settings and custom CSV separators or terminators.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, mysql
- Domain
- database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100