Bug:windows环境下配置hdfsWriter,由于windows文件分隔符为\,会导致写完hdfs删除临时目录的时候,误删数据库
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
dataX日志如下:
09:30:26.485 [job-0] INFO c.a.d.p.w.hdfswriter.HdfsWriter$Job - start rename file [hdfs://hacluster/user/hive/warehouse/ods.db/test_maozl__504dceda_c549_49c5_b914_cf6483664813\file__b1d2c12d_4bf2_494c_8e1e_6f4be4fd8b37] to file [hdfs://hacluster/user/hive/warehouse/ods.db/test_maozl\file__b1d2c12d_4bf2_494c_8e1e_6f4be4fd8b37].
09:30:26.524 [job-0] INFO c.a.d.p.w.hdfswriter.HdfsWriter$Job - finish rename file [hdfs://hacluster/user/hive/warehouse/ods.db/test_maozl__504dceda_c549_49c5_b914_cf6483664813\file__b1d2c12d_4bf2_494c_8e1e_6f4be4fd8b37] to file [hdfs://hacluster/user/hive/warehouse/ods.db/test_maozl\file__b1d2c12d_4bf2_494c_8e1e_6f4be4fd8b37].
09:30:26.524 [job-0] INFO c.a.d.p.w.hdfswriter.HdfsWriter$Job - start delete tmp dir [hdfs://hacluster/user/hive/warehouse/ods.db].
09:30:26.551 [job-0] INFO c.a.d.p.w.hdfswriter.HdfsWriter$Job - finish delete tmp dir [hdfs://hacluster/user/hive/warehouse/ods.db].
通过调试发现,是由于临时文件的目录里包含windows的文件分隔符\,导致删除临时文件的时候,hadoop的Path.getParent()找到的路径为hdfs://hacluster/user/hive/warehouse/ods.db,进而直接把数据库给删掉了。
修改方法:
将HdfsWriter里的文件分割符替换为IOUtils.DIR_SEPARATOR_UNIX
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in HdfsWriter by tracing how temporary file paths are assembled on Windows and how Path.getParent() is used during temporary-directory deletion. Compare the logged paths with the reported deletion behavior, then verify that using IOUtils.DIR_SEPARATOR_UNIX prevents the parent HDFS database directory from being selected for deletion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100