[Feature][EngineConn] Enable customized and isolated python environment for Pyspark
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-linkis/issues) and found no similar feature requirement.
### Problem Description
For Python developers, using a customized and isolated Python runtime environment is an indispensable requirement.
You and your colleagues may want to use different versions of python and python packages and don’t want to affect each others’ environment.
Thus it's necessary to support using customized and isolated Python environment in hadoop yarn cluster and achieve this for PySpark.
### Description
_No response_
### Use case
_No response_
### solutions
From the official [Spark Configuration](https://spark.apache.org/docs/latest/configuration.html), we can know that
Property `spark.pyspark.driver.python` take precedence than `PYSPARK_DRIVER_PYTHON` if it is set.
Property `spark.pyspark.python` take precedence than `PYSPARK_PYTHON` if it is set.
And Property `archives ` is the same as `spark.yarn.dist.archives`.
We can leverage the global settings module, to let user define the variables.
By default, these variables can be empty, and in that case EnvVar in `spark-defaults.conf` or `spark-env.sh` will take precedence.
If user defines, the defined variables will be isolated from other users, and take precedence.
They will be appended into the Spark engine start script, when them can be read to overwrite the default setting in `spark-defaults.conf` or `spark-env.sh`.
### Anything else
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
Contributor guide
Assessment
This issue has not been assessed yet.