alibaba / alibaba/DataX

在window系统上本地调试,使用hdfswriter插件后会导致写入目录被删除

Open
#1,048 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

### 问题描述:
使用hdfswriter向hive写入数据时,发现在数据写入到hdfs中的文件成功后,会将该hive表所在的库对应的目录删除掉,
比如hive库名为aaaa,表名为dj_tzfxx_test,则再向/warehouse/tablespace/managed/hive/aaaa.db/dj_tzfxx_test目录下写入数据后,
会将aaaa.db目录删除掉。

**hdfswriter创建临时目录源代码**:
`/**
* 创建临时目录
* @param userPath
* @return
*/
private String buildTmpFilePath(String userPath) {
String tmpFilePath;
boolean isEndWithSeparator = false;
switch (IOUtils.DIR_SEPARATOR) {
case IOUtils.DIR_SEPARATOR_UNIX:
isEndWithSeparator = userPath.endsWith(String
.valueOf(IOUtils.DIR_SEPARATOR));
break;
case IOUtils.DIR_SEPARATOR_WINDOWS:
isEndWithSeparator = userPath.endsWith(String
.valueOf(IOUtils.DIR_SEPARATOR_WINDOWS));
break;
default:
break;
}
String tmpSuffix;
tmpSuffix = UUID.randomUUID().toString().replace('-', '_');
if (!isEndWithSeparator) {
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR);
}else if("/".equals(userPath)){
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR);
}else{
tmpFilePath = String.format("%s__%s%s", userPath.substring(0,userPath.length()-1), tmpSuffix, IOUtils.DIR_SEPARATOR);
}```

### 问题分析:
由于是采用的本地调试,属于window系统,dataX在生成临时目录时,使用的是反斜杠‘\’来连接,
导致在dataX识别不出来这是目录分隔符,导致获取父级目录时出错,原本的父级目录应该是dj_tzfxx_test5__83176204_1a35_4249_b91f_ce2403b78c66,
由于识别不出来反斜杠'\',导致获取到的父级目录为aaaa.db,从而在删除临时目录时,将aaaa.db作为临时目录删除了。

**代码调整:**
` /**
* 创建临时目录
*
* @param userPath
* @return
*/
private String buildTmpFilePath(String userPath) {
String tmpFilePath;
boolean isEndWithSeparator = false;
switch (IOUtils.DIR_SEPARATOR) {
case IOUtils.DIR_SEPARATOR_UNIX:
isEndWithSeparator = userPath.endsWith(String
.valueOf(IOUtils.DIR_SEPARATOR));
break;
case IOUtils.DIR_SEPARATOR_WINDOWS:
isEndWithSeparator = userPath.endsWith(String
.valueOf(IOUtils.DIR_SEPARATOR_WINDOWS));
break;
default:
break;
}
String tmpSuffix;
tmpSuffix = UUID.randomUUID().toString().replace('-', '_');
if (!isEndWithSeparator) {
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR_UNIX);
} else if ("/".equals(userPath)) {
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR);
} else {
// tmpFilePath = String.format("%s__%s%s", userPath.substring(0,userPath.length()-1), tmpSuffix, IOUtils.DIR_SEPARATOR);
// 适配window系统目录拼接 warehouse/tablespace/managed/hive/aaaa.db/dj_tzfxx_test5__dee1d1ea_dbfc_4a7b_aa5a_6783a7b97bcf/
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR_UNIX);
}
while (hdfsHelper.isPathexists(tmpFilePath)) {
tmpSuffix = UUID.randomUUID().toString().replace('-', '_');
if (!isEndWithSeparator) {
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR);
} else if ("/".equals(userPath)) {
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR);
} else {
// tmpFilePath = String.format("%s__%s%s", userPath.substring(0,userPath.length()-1), tmpSuffix, IOUtils.DIR_SEPARATOR);
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR_UNIX);
}
}
return tmpFilePath;
}```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the hdfswriter buildTmpFilePath method and inspect how IOUtils.DIR_SEPARATOR is used when constructing temporary HDFS paths on Windows. Reproduce the write and cleanup flow with a path ending in a separator; done means the temporary directory uses the intended HDFS separator and cleanup leaves the Hive database directory intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
databases, operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.