HdfsWriter delete wrong files when execute on Windows. Windows机器下执行HDFS写入存在删数据的问题。
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
在Windows机器下写入到HDFS时,有严重bug,会导致指定path下的所有数据删除,附上同一个json配置在windows和Linux下执行的日志:
### windows下的DataX执行日志:
2018-06-21 18:08:52.665 [job-0] INFO HdfsWriter$Job - start delete tmp dir [hdfs://172.30.120.15:8020/inceptor1/user/hive/warehouse/default.db/hive] .
2018-06-21 18:08:52.761 [job-0] INFO HdfsWriter$Job - finish delete tmp dir [hdfs://172.30.120.15:8020/inceptor1/user/hive/warehouse/default.db/hive] .
### linux下的DataX执行日志:
2018-06-21 16:42:43.633 [job-0] INFO HdfsWriter$Job - start delete tmp dir [hdfs://172.30.120.15:8020/inceptor1/user/hive/warehouse/default.db/hive/test__b4800119_af56_4cf2_b4e4_af49751f69ac] .
2018-06-21 16:42:43.642 [job-0] INFO HdfsWriter$Job - finish delete tmp dir [hdfs://172.30.120.15:8020/inceptor1/user/hive/warehouse/default.db/hive/test__b4800119_af56_4cf2_b4e4_af49751f69ac] .
看了下源码,是在设置文件路径分隔符时,选用的是DataX运行环境的文件分隔符,而不是HDFS运行环境的文件分隔符。
源码路径:com/alibaba/datax/plugin/writer/hdfswriter/HdfsWriter.java
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading com/alibaba/datax/plugin/writer/hdfswriter/HdfsWriter.java and tracing how the temporary HDFS deletion path is assembled on Windows versus Linux. Compare the logged paths with the configured target; done means Windows deletes only the generated temporary directory, matching the Linux behavior, rather than the configured base path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100