apache / apache/kyuubi

[Bug] spark engine status retrieval fails if `master` is set

Open
#6,647 1 comment 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

# current set up
I am running kyuubi 1.9.1 on AKS, clients submit `batch` jobs using the kyuubi API. I have set the `KUBECONFIG` envvar to the path to a context file containing the contexts for my spark worker clusters (I have dedicated K8s for spark jobs).

## Server config:
Kyuubi server config are as follows:
```
kyuubi.kubernetes.isost10.spark.authenticate.oauthTokenFile=/etc/isost10/token
kyuubi.kubernetes.isost9.spark.authenticate.oauthTokenFile=/etc/isost9/token
```

## Clients sample request
User are able are able to post the following:
```JSON
{
"resource": "local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar",
"name": "sample-job",
"batchType": "SPARK",
"className": "org.apache.spark.examples.SparkPi",
"conf": {
"spark.kubernetes.context": "isost9",
"spark.master": "k8s://cluster:443",
"spark.kubernetes.container.image": "acr.io/spark:3.4.1-5250722",
"spark.kubernetes.namespace": "spark",
"spark.kubernetes.serviceAccountName": "spark",
"spark.kubernetes.driver.node.selector.label": "nodepool1",
"spark.kubernetes.executor.node.selector.label": "nodepool1",
"spark.executor.memory": "4G",
"spark.executor.cores": "2",
"spark.driver.memory": "4G",
"spark.driver.cores": "2"
}
}
```

# The problem
The issue occurs when we set the master on the kyuubi side per context:
```
kyuubi.kubernetes..master.address=k8s://cluster:443
kyuubi.kubernetes...authenticate.oauthTokenFile
```

The `batch` job get scheduled in the remote cluster, but kyuubi is unable to retrieve the status. The logs will contain:
```
ERROR OkHttp http://k8s/... io.fabric8.kubernetes.client.informers.impl.cache.Reflector: listSyncAndWatch failed for v1/namespaces/spark/pods, will stop
java.util.concurrent.CompletionException: java.net.UnknownHostException: k8s
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$1.onFailure(OkHttpClientImpl.java:330)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:211)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.net.UnknownHostException: k8s
```

additionally clients are *REQUIRED* to set the spark master.

Is it supported to set spark-master per context, so the clients doesn't have to know the API server IP of the backend cluster?

### Affects Version(s)

1.9.1

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

_No response_

### Kyuubi Server Configurations

_No response_

### Kyuubi Engine Configurations

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [x] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing a Kyuubi API batch submission with kyuubi.kubernetes..master.address set, then trace status retrieval and the Kubernetes client endpoint from the reported UnknownHostException. Done means the per-context master behavior is resolved or confirmed unsupported, with status retrieval verified for the configuration shown.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, spark
Domain
cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.