kubeflow / kubeflow/spark-operator
Encountering error "argument list too long" while starting processes in pods, due to too many environment variables for k8s services "*-ui-svc"
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 1.5k
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 13
Description
### What happened?
after having launched many (>=2000) SparkApplication(s) in my cluster, then error occurs for all new processes in mmy entrpoints on new pods, saying ERROR "argument list too long"
for example a simple pod doing "curl http://github.com" will stop working, with error "argument list too long"
After investigating, it appears that my sparkApplications are all COMPLETED, I have very few still "RUNNING"
All the kubernetes service(s) corresponding to finished "${mySparkApp}-svc" have been successfully removed,
BUT all the other kubernetes service(s) "${mySparkApp}-ui-svc" are STILL present in kubernetes
For each zombi service (not corresponding to a running service / sparkApplication) any more, I still see several environement variable created for each new pod on my cluster:
export | grep UI_SVC_SERVICE_
declare -x SPARKAPP1_JAVA_UI_SVC_SERVICE_PORT="4040"
declare -x SPARKAPP1_JAVA_UI_SVC_SERVICE_HOST="10.0.96.1"
declare -x SPARKAPP2_JAVA_UI_SVC_SERVICE_PORT="4040"
declare -x SPARKAPP2_JAVA_UI_SVC_SERVICE_HOST="10.0.96.2"
.... tons of env vars
declare -x SPARKAPP2000_JAVA_UI_SVC_SERVICE_PORT="4040"
declare -x SPARKAPP2000_JAVA_UI_SVC_SERVICE_HOST="10.0.98.253"
### Reproduction Code
_No response_
### Expected behavior
_No response_
### Actual behavior
_No response_
### Environment & Versions
- Kubernetes Version:
- Spark Operator Version:
- Apache Spark Version:
### Additional context
_No response_
### Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
Contributor guide
Research direction
Start by reproducing the issue with many completed SparkApplications and inspect the cleanup path for the regular service versus the '*-ui-svc' services. Record the Kubernetes, Spark Operator, and Apache Spark versions, then verify that completed applications remove their UI services and that new pods no longer inherit the accumulated environment variables.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python, spark
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100