kubeflow / kubeflow/spark-operator

Encountering error "argument list too long" while starting processes in pods, due to too many environment variables for k8s services "*-ui-svc"

Open
#2,984 3 comments 0 reactions 0 assignees View on GitHub
kind/bug
Dominant language
Python
Stars
3.2k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
13

Description

### What happened?

after having launched many (>=2000) SparkApplication(s) in my cluster, then error occurs for all new processes in mmy entrpoints on new pods, saying ERROR "argument list too long"
for example a simple pod doing "curl http://github.com" will stop working, with error "argument list too long"

After investigating, it appears that my sparkApplications are all COMPLETED, I have very few still "RUNNING"
All the kubernetes service(s) corresponding to finished "${mySparkApp}-svc" have been successfully removed,
BUT all the other kubernetes service(s) "${mySparkApp}-ui-svc" are STILL present in kubernetes

For each zombi service (not corresponding to a running service / sparkApplication) any more, I still see several environement variable created for each new pod on my cluster:

export | grep UI_SVC_SERVICE_
declare -x SPARKAPP1_JAVA_UI_SVC_SERVICE_PORT="4040"
declare -x SPARKAPP1_JAVA_UI_SVC_SERVICE_HOST="10.0.96.1"
declare -x SPARKAPP2_JAVA_UI_SVC_SERVICE_PORT="4040"
declare -x SPARKAPP2_JAVA_UI_SVC_SERVICE_HOST="10.0.96.2"
.... tons of env vars
declare -x SPARKAPP2000_JAVA_UI_SVC_SERVICE_PORT="4040"
declare -x SPARKAPP2000_JAVA_UI_SVC_SERVICE_HOST="10.0.98.253"

### Reproduction Code

_No response_

### Expected behavior

_No response_

### Actual behavior

_No response_

### Environment & Versions

- Kubernetes Version:
- Spark Operator Version:
- Apache Spark Version:

### Additional context

_No response_

### Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with many completed SparkApplications and inspect the cleanup path for the regular service versus the '*-ui-svc' services. Record the Kubernetes, Spark Operator, and Apache Spark versions, then verify that completed applications remove their UI services and that new pods no longer inherit the accumulated environment variables.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python, spark
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.