[Bug] spark.authenticate.secret is not picked up in spark standalone with cluster mode
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
Hi,
Thanks for a great program!
When running a spark standalone cluster with security enable, it fails with:
```
java.lang.IllegalArgumentException: requirement failed: A secret key must be specified via the spark.authenticate.secret config.
```
though it works with client mode.
I believe the issue is this: https://issues.apache.org/jira/browse/SPARK-25078 (marked as resolved, but it's not according to the last comment.)
Do you think it would be possible to implement that workaround described there?
### Affects Version(s)
1.11.0
### Kyuubi Server Log Output
```logtalk
```
### Kyuubi Engine Log Output
```logtalk
```
### Kyuubi Server Configurations
```yaml
```
### Kyuubi Engine Configurations
```yaml
```
### Additional context
_No response_
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Research direction
Start by reproducing the failure with Spark standalone in cluster mode and compare it with the working client-mode path. Read the workaround described in SPARK-25078 and trace how Kyuubi passes spark.authenticate.secret; done means the secret is picked up in cluster mode without regressing client mode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100