DTStack / DTStack/Taier

[Bug] [Flink Jar] Download hdfs jar fail when running flink jar task:xxxx is not a valid DFS filename

Open
#1,098 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.3k
Forks
350
PR merge metrics
No merged PRs in 30d

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/DTStack/Taier/issues) and found no similar issues.

### What happened

When running flink jar task with a resource which is stored in hdfs,taier can not download jar from hdfs successfully.And it makes flink jobs run fail.
![图片](https://github.com/DTStack/Taier/assets/54771054/6307d2f8-c8e4-41df-a8c1-c4d88d6b1555)

### What you expected to happen

I have seen the source code and debug,found that the reason for this problem is related to `StreamSqlFormatService#generateAddJarSql(String,String)` and `FileSystemManager#downloadFile(...)`

`StreamSqlFormatService` will replace hdfs://localhost:9820 to hdfs:/localhost:9820,

When I return resource url directly, here comes the second question:
`FileSystemManager` will remove prefix(hdfs://), which make url to `localhost:9820`.Of course,`HdfsFileManage` download fail

### How to reproduce

follow below steps can produce this problem:

1. upload a flink application jar and choose hdfs storage
![resource-info](https://github.com/DTStack/Taier/assets/54771054/420dd576-2a35-4116-aae5-7ccd46bdf9da)
2. create flink jar task and run.

### Anything else

_No response_

### Version

master

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.