dumpling: support encoding binary data in output
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
Some tools that ingest CSV output (in particular Snowflake) cannot handle raw binary data in CSV. They require it to be encoded using a scheme such as base64 or hex.
**Describe the feature you'd like:**
dumpling should support encoding all binary data types using either base64 or hex. dumpling can automatically identify the data types of columns by looking at information_schema.
It may also be helpful if dumpling can be configured to encode all string-type columns, to accommodate data encoded in character encodings other than utf8.
The specific encodings supported should be based on an analysis of various tools that may consume CSV.
It may also be possible for dumpling to support a flexible encoding scheme based on SQL functions. If a user wants to dump data as hex-encoded, for example, dumpling could support something like `--encode-binary-function='HEX(%s)'`.
**Describe alternatives you've considered:**
Other solutions may include using the `--sql` argument to dumpling, but that is very difficult to use and it prevents dumpling from dumping in parallel.
**Teachability, Documentation, Adoption, Migration Strategy:**
Contributor guide
Assessment
This issue has not been assessed yet.