apache / apache/kyuubi

[Bug] spark.authenticate.secret is not picked up in spark standalone with cluster mode

Open
#7,366 3 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

Hi,

Thanks for a great program!

When running a spark standalone cluster with security enable, it fails with:
```
java.lang.IllegalArgumentException: requirement failed: A secret key must be specified via the spark.authenticate.secret config.
```
though it works with client mode.

I believe the issue is this: https://issues.apache.org/jira/browse/SPARK-25078 (marked as resolved, but it's not according to the last comment.)

Do you think it would be possible to implement that workaround described there?

### Affects Version(s)

1.11.0

### Kyuubi Server Log Output

```logtalk

```

### Kyuubi Engine Log Output

```logtalk

```

### Kyuubi Server Configurations

```yaml

```

### Kyuubi Engine Configurations

```yaml

```

### Additional context

_No response_

### Are you willing to submit PR?

- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with Spark standalone in cluster mode and compare it with the working client-mode path. Read the workaround described in SPARK-25078 and trace how Kyuubi passes spark.authenticate.secret; done means the secret is picked up in cluster mode without regressing client mode.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.