在window系统上本地调试,使用hdfswriter插件后会导致写入目录被删除
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
### 问题描述:
使用hdfswriter向hive写入数据时,发现在数据写入到hdfs中的文件成功后,会将该hive表所在的库对应的目录删除掉,
比如hive库名为aaaa,表名为dj_tzfxx_test,则再向/warehouse/tablespace/managed/hive/aaaa.db/dj_tzfxx_test目录下写入数据后,
会将aaaa.db目录删除掉。
**hdfswriter创建临时目录源代码**:
`/**
* 创建临时目录
* @param userPath
* @return
*/
private String buildTmpFilePath(String userPath) {
String tmpFilePath;
boolean isEndWithSeparator = false;
switch (IOUtils.DIR_SEPARATOR) {
case IOUtils.DIR_SEPARATOR_UNIX:
isEndWithSeparator = userPath.endsWith(String
.valueOf(IOUtils.DIR_SEPARATOR));
break;
case IOUtils.DIR_SEPARATOR_WINDOWS:
isEndWithSeparator = userPath.endsWith(String
.valueOf(IOUtils.DIR_SEPARATOR_WINDOWS));
break;
default:
break;
}
String tmpSuffix;
tmpSuffix = UUID.randomUUID().toString().replace('-', '_');
if (!isEndWithSeparator) {
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR);
}else if("/".equals(userPath)){
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR);
}else{
tmpFilePath = String.format("%s__%s%s", userPath.substring(0,userPath.length()-1), tmpSuffix, IOUtils.DIR_SEPARATOR);
}```
### 问题分析:
由于是采用的本地调试,属于window系统,dataX在生成临时目录时,使用的是反斜杠‘\’来连接,
导致在dataX识别不出来这是目录分隔符,导致获取父级目录时出错,原本的父级目录应该是dj_tzfxx_test5__83176204_1a35_4249_b91f_ce2403b78c66,
由于识别不出来反斜杠'\',导致获取到的父级目录为aaaa.db,从而在删除临时目录时,将aaaa.db作为临时目录删除了。
**代码调整:**
` /**
* 创建临时目录
*
* @param userPath
* @return
*/
private String buildTmpFilePath(String userPath) {
String tmpFilePath;
boolean isEndWithSeparator = false;
switch (IOUtils.DIR_SEPARATOR) {
case IOUtils.DIR_SEPARATOR_UNIX:
isEndWithSeparator = userPath.endsWith(String
.valueOf(IOUtils.DIR_SEPARATOR));
break;
case IOUtils.DIR_SEPARATOR_WINDOWS:
isEndWithSeparator = userPath.endsWith(String
.valueOf(IOUtils.DIR_SEPARATOR_WINDOWS));
break;
default:
break;
}
String tmpSuffix;
tmpSuffix = UUID.randomUUID().toString().replace('-', '_');
if (!isEndWithSeparator) {
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR_UNIX);
} else if ("/".equals(userPath)) {
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR);
} else {
// tmpFilePath = String.format("%s__%s%s", userPath.substring(0,userPath.length()-1), tmpSuffix, IOUtils.DIR_SEPARATOR);
// 适配window系统目录拼接 warehouse/tablespace/managed/hive/aaaa.db/dj_tzfxx_test5__dee1d1ea_dbfc_4a7b_aa5a_6783a7b97bcf/
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR_UNIX);
}
while (hdfsHelper.isPathexists(tmpFilePath)) {
tmpSuffix = UUID.randomUUID().toString().replace('-', '_');
if (!isEndWithSeparator) {
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR);
} else if ("/".equals(userPath)) {
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR);
} else {
// tmpFilePath = String.format("%s__%s%s", userPath.substring(0,userPath.length()-1), tmpSuffix, IOUtils.DIR_SEPARATOR);
tmpFilePath = String.format("%s__%s%s", userPath, tmpSuffix, IOUtils.DIR_SEPARATOR_UNIX);
}
}
return tmpFilePath;
}```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the hdfswriter buildTmpFilePath method and inspect how IOUtils.DIR_SEPARATOR is used when constructing temporary HDFS paths on Windows. Reproduce the write and cleanup flow with a path ending in a separator; done means the temporary directory uses the intended HDFS separator and cleanup leaves the Hive database directory intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- databases, operating-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100