alibaba / alibaba/DataX

关于读取文件对null值和空串的处理

Open
#2,094 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

datax在读取文件内容时,由InputStream到BufferReader到javacsv提供CsvReader,转换成CsvReader时,两分隔符之间的空值都被识别为空串,比如“1”,,"" 想被识别为“1”,null,"" ,但实际是识别为“1”,“”,“” 配置nullformat不能满足要求,因为这里是需要区分空值和空串,我这边的做法是将javacsv替换成opencsv,使用opencsv的CSVReader,该类提供了区分null和空串的方法,也能识别两引号之间的分隔符,但是无法评估其他可能的影响,这里想请教一下使用javacsv的理由是什么,有没有不可替代性,查看中央仓库发现javacsv是2008年的包,只有一个版本且后面从未更新过,是否可以替换为opencsv,这里贴出修改的地方
![image](https://github.com/alibaba/DataX/assets/66933135/7cdb1713-0e13-49c5-9360-10b0ff4d286c)
![image](https://github.com/alibaba/DataX/assets/66933135/b5a77ea5-fd05-4dd7-97af-a6e6c1bde288)
![image](https://github.com/alibaba/DataX/assets/66933135/e8fe7ff4-d28c-49a4-889a-8501cc732e5c)
![image](https://github.com/alibaba/DataX/assets/66933135/f29ab43f-cdde-42a1-90b3-d2b1ca2d6f4d)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the file-reading path described from InputStream through BufferedReader to javacsv's CsvReader, then compare its handling with opencsv's CSVReader. Confirm how null fields, empty fields, and quoted delimiters are represented; done means preserving the intended distinction while checking for other parsing regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.