kubeflow / kubeflow/spark-operator
Deal with internally created ConfigMap mounted onto SPARK_CONF_DIR
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 1.5k
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 13
Description
PR apache/spark#20669 introduced a ConfigMap carrying Spark configuration properties in a file for the driver. The environment variable SPARK_CONF_DIR is set to point to the mount path /opt/spark/conf of the ConfigMap. This is in conflict with what spec.sparkConfigMap is designed to do. We need to find a solution to work with the internally created and mounted ConfigMap.
Contributor guide
Research direction
Start by reading PR apache/spark#20669 and tracing how SPARK_CONF_DIR, the internally created ConfigMap, and spec.sparkConfigMap are handled in the operator. Done means the internal mount and user-configured spec.sparkConfigMap no longer conflict, with behavior validated against the relevant operator flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, spark
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100