pingcap / pingcap/tidb

lightning: process csv as character stream

Open
#33,830 3 comments 0 reactions 0 assignees View on GitHub
component/lightning type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
currently we process csv file as byte stream, search `backslash/delimiter/separator` in it, but some characters may contain those special characters, such as `穃` in gbk is `c1 d6 b7 5c` contains `\(5c)`, lightning will report error or misinterpret values.

we can process csv as character stream to avoid such problem.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.