alibaba / alibaba/DataX

HDFS Writer String 类型 中存在 '\n'

Open
#71 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

当数据中存在 \n 时写入的数据会换行、、由于hive、impala等默认是 换行符 分割 每条数据,所以会导致 这部分数据 错乱,可不可以考虑在写入的时候 replace 成 空格呢,我遇到的情况是 直接存 html ,其中的 \n 导致的

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the HDFS Writer path that serializes String values, then reproduce the issue with HTML containing newline characters and check how Hive or Impala reads the resulting records. Confirm the intended newline handling and add or update coverage so the output no longer splits the value into separate records.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.