DTStack / DTStack/chunjun

[Bug] 数据同步remoteChunJunDistDir 参数不生效

Open
#1,740 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.1k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar issues.

### What happened

一: 参数设置如下
argsList.add("-remoteChunJunDistDir");
argsList.add("hdfs://nameHAservice/user/badmin/ruitest/usrlib/chunjun-dist");

image

即使我填写是hdfs地址,但是这段远程代码只能判断本地的jar是否存在,而不能判断远程的jar是否存在。导致后续一系列的问题

二:后面我通过设置yarn.provided.usrlib.dir 参数来解决这个无法配置远程chunjun包的问题
stringBuilder
.append("{")
.append("\"execution.target\":\"yarn-per-job\"")
.append(", ")
.append("\"yarn.provided.usrlib.dir\":\"/user/badmin/ruitest/usrlib\"")
.append(", ")
.append("\"yarn.provided.lib.dirs\":\"/user/badmin/ruitest/flink-1.16.1/lib\"")
.append(", ")
.append("\"yarn.application.queue\":\"root.root\"")
.append("} ");

image
但是上面的这段代码依然会导致将版本的chunjun connector包上传到hdfs,最后我将该段代码注释了。解决了这个问题。

希望remoteChunJunDistDir参数能够真实起作用

### What you expected to happen

remoteChunJunDistDir设置后,可以使用hdfs上的chunjun jar

### How to reproduce

参数设置如上

### Anything else

_No response_

### Version

master

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.