alibaba / alibaba/DataX

Bug:windows环境下配置hdfsWriter,由于windows文件分隔符为\,会导致写完hdfs删除临时目录的时候,误删数据库

Open
#458 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

dataX日志如下:
09:30:26.485 [job-0] INFO  c.a.d.p.w.hdfswriter.HdfsWriter$Job - start rename file [hdfs://hacluster/user/hive/warehouse/ods.db/test_maozl__504dceda_c549_49c5_b914_cf6483664813\file__b1d2c12d_4bf2_494c_8e1e_6f4be4fd8b37] to file [hdfs://hacluster/user/hive/warehouse/ods.db/test_maozl\file__b1d2c12d_4bf2_494c_8e1e_6f4be4fd8b37].
09:30:26.524 [job-0] INFO  c.a.d.p.w.hdfswriter.HdfsWriter$Job - finish rename file [hdfs://hacluster/user/hive/warehouse/ods.db/test_maozl__504dceda_c549_49c5_b914_cf6483664813\file__b1d2c12d_4bf2_494c_8e1e_6f4be4fd8b37] to file [hdfs://hacluster/user/hive/warehouse/ods.db/test_maozl\file__b1d2c12d_4bf2_494c_8e1e_6f4be4fd8b37].
09:30:26.524 [job-0] INFO  c.a.d.p.w.hdfswriter.HdfsWriter$Job - start delete tmp dir [hdfs://hacluster/user/hive/warehouse/ods.db].
09:30:26.551 [job-0] INFO  c.a.d.p.w.hdfswriter.HdfsWriter$Job - finish delete tmp dir [hdfs://hacluster/user/hive/warehouse/ods.db].

通过调试发现,是由于临时文件的目录里包含windows的文件分隔符\,导致删除临时文件的时候,hadoop的Path.getParent()找到的路径为hdfs://hacluster/user/hive/warehouse/ods.db,进而直接把数据库给删掉了。

修改方法:
将HdfsWriter里的文件分割符替换为IOUtils.DIR_SEPARATOR_UNIX

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in HdfsWriter by tracing how temporary file paths are assembled on Windows and how Path.getParent() is used during temporary-directory deletion. Compare the logged paths with the reported deletion behavior, then verify that using IOUtils.DIR_SEPARATOR_UNIX prevents the parent HDFS database directory from being selected for deletion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.