kyuubi Is it possible to support spark on yarn SharedCacheClient
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
It will not be shared by nodemagner, which will result in a large number of file downloads and deletions, causing significant consumption of disk I/O. Currently, Yarn has a SharedCache service, but Spark and Kyuubi do not have related integrations. Can they be integrated?
### Motivation
spark submit job example:
sun.java.command=org.apache.spark.deploy.yarn.ApplicationMaster --class org.apache.kyuubi.engine.spark.SparkSQLEngine --jar file:/data/services/kyuubi_package-t10141512.56c0a726.r/apache-kyuubi-1.7.1-bin/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.7.1.jar ...
kyuubi-spark-sql-engine_2.12-1.7.1.jar
### Describe the solution
Realize the following functions
https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/SharedCache.html
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Assessment
This issue has not been assessed yet.