apache / apache/livy

Spark on Kubernetes job fails if service account doesn't have permission to all namespaces.

Open
#461 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
963
Forks
625
Avg merge
6d 21h
Merged PRs (30d)
6

Description

In a multi tenant Kubernetes cluster, it will not always be the case that livy service account has access to all namespaces. Since Livy currently looks for job in all namespaces, it fails if the permission is not there for some namespace in cluster.

Ideally we should only look for job in the namespace it was submitted to. Relevant code is here, i think : https://github.com/apache/incubator-livy/blob/1f6bd7ab55db3e5b08ff133d554b32db83a23890/server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala#L446
Also this,
https://github.com/apache/incubator-livy/blob/1f6bd7ab55db3e5b08ff133d554b32db83a23890/server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala#L689

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala at the references around lines 446 and 689. Trace how Livy searches for the submitted Spark job across namespaces and how the submission namespace is available. Done means a multi-tenant Kubernetes setup no longer requires the Livy service account to access unrelated namespaces.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, scala, spark
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.