alibaba / alibaba/DataX

windows环境下hdsfwriter组件中存在取父路径的问题

Open
#200 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

环境介绍:
1.window10环境下JDK1.8调试状态。
问题介绍:
hdfswriter.java中第214行获取临时数据存放路径后,由于出现反斜杠'\',导致路径在获取父目录用于删除临时文件时获取错误的父目录,将‘\’当成了文件名称,而不是目录层级标识,最终错误的删除了我指定的整个目录,同时最终数据因为删除了更上级的父目录,导致传输至最终目的的数据也被删除。
暂时的修改方式是:
判断storePath和endStorePath是否包含'/',如果包含,则将'\'替换成/。
经过测试,最终在MAC,Linux,windows测试通过。
` String storePath = buildTmpFilePath(this.path);
if(storePath != null && storePath.contains("/")){
//由于在window上调试获取的路径为转义字符代表的斜杠,故出现被认为是文件名称的一部分,使得获取父目录存在问题,最终影响到直接删除更高一层的目录,导致Hive数据出现问题。
storePath = storePath.replace('\\', '/');
}
//最终存放路径
String endStorePath = buildFilePath();
if(endStorePath != null && endStorePath.contains("/")){
//storePath.replaceAll("\\", "/");
endStorePath = endStorePath.replace('\\', '/');
}`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in hdfswriter.java around line 214 and inspect how storePath and endStorePath are built and how their parent directories are selected for temporary-file deletion. Reproduce the Windows JDK 1.8 behavior, then verify that parent resolution and cleanup preserve the intended directory and final transferred data on Windows, macOS, and Linux.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.