alibaba / alibaba/DataX

HdfsWriter delete wrong files when execute on Windows. Windows机器下执行HDFS写入存在删数据的问题。

Open
#109 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

在Windows机器下写入到HDFS时,有严重bug,会导致指定path下的所有数据删除,附上同一个json配置在windows和Linux下执行的日志:

### windows下的DataX执行日志:
2018-06-21 18:08:52.665 [job-0] INFO HdfsWriter$Job - start delete tmp dir [hdfs://172.30.120.15:8020/inceptor1/user/hive/warehouse/default.db/hive] .
2018-06-21 18:08:52.761 [job-0] INFO HdfsWriter$Job - finish delete tmp dir [hdfs://172.30.120.15:8020/inceptor1/user/hive/warehouse/default.db/hive] .

### linux下的DataX执行日志:
2018-06-21 16:42:43.633 [job-0] INFO HdfsWriter$Job - start delete tmp dir [hdfs://172.30.120.15:8020/inceptor1/user/hive/warehouse/default.db/hive/test__b4800119_af56_4cf2_b4e4_af49751f69ac] .
2018-06-21 16:42:43.642 [job-0] INFO HdfsWriter$Job - finish delete tmp dir [hdfs://172.30.120.15:8020/inceptor1/user/hive/warehouse/default.db/hive/test__b4800119_af56_4cf2_b4e4_af49751f69ac] .

看了下源码,是在设置文件路径分隔符时,选用的是DataX运行环境的文件分隔符,而不是HDFS运行环境的文件分隔符。
源码路径:com/alibaba/datax/plugin/writer/hdfswriter/HdfsWriter.java

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading com/alibaba/datax/plugin/writer/hdfswriter/HdfsWriter.java and tracing how the temporary HDFS deletion path is assembled on Windows versus Linux. Compare the logged paths with the configured target; done means Windows deletes only the generated temporary directory, matching the Linux behavior, rather than the configured base path.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.