kubeflow / kubeflow/spark-operator

Fix Spark Connect container argumenets construction

Open
#3,072 8 comments 2 reactions 0 assignees View on GitHub
kind/bug
Dominant language
Python
Stars
3.2k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
13

Description

### What happened?

Currently Spark Connect controller constructs container command and args with

```
container.Command = []string{"bash", "-c"}
args, err := buildStartConnectServerArgs(conn)
if err != nil {
return fmt.Errorf("failed to build spark connection args: %v", err)
}
container.Args = []string{strings.Join(args, " ")}
```

This is not ideal because everything is wrapped in `bash -c` which means all values passed in with the arg field are subject to shell interpretation.

We need to follow the way Spark Application Controller does it here: https://github.com/jayanth86/spark-operator/blob/00ba077967057560ad34949906d5ff4846659ce2/internal/controller/sparkapplication/submission.go#L71-L94

### Reproduction Code

_No response_

### Expected behavior

_No response_

### Actual behavior

_No response_

### Environment & Versions

- Kubernetes Version:
- Spark Operator Version:
- Apache Spark Version:

### Additional context

_No response_

### Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Contributor guide

Open the contributing guide

Research direction

Locate the Spark Connect controller code that calls buildStartConnectServerArgs and compare its container command and argument construction with internal/controller/sparkapplication/submission.go from the referenced commit. Confirm that arguments are passed without bash -c or shell joining, and run the relevant controller tests to verify Spark Connect startup still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.