kubeflow / kubeflow/sdk

SparkClient never turns on the Spark UI or metrics for a submitted job

Open
#733 6 comments 0 reactions 1 assignee Claimed by @Talha8899 View on GitHub
area/observability area/spark kind/feature
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

Hi, I was digging through the SparkClient code and noticed something.

Spark itself comes with a built in dashboard that shows you what a job is doing while it runs and after it finishes, which steps are done, which one is stuck, how long each part took. It also supports sending out Prometheus metrics through a sidecar. Both of these are just fields on the SparkApplication spec, `sparkUIOptions` and `monitoring`, that the operator already understands and supports.

I checked both places where SparkClient builds a SparkApplication, `get_spark_application_cr_from_file_job` and `get_spark_application_cr_from_func_job` in `kubeflow/spark/backends/kubernetes/utils.py`, and neither field ever gets set. I also checked the public SparkClient API in `spark_client.py` and there is nothing there about the UI or metrics either.

So right now if you submit a job with `submit_job()` and it hangs or runs slow, the only thing you can look at is raw logs through `get_job_logs()`. There is no way to actually see the job, even though Spark already has that dashboard built and ready, this SDK just never connects it.

I would like to work on this. My plan is to add an option to `submit_job()` that turns on the UI (something like `enable_ui`) and wires up `spark_ui_options` with a sensible default, plus a small helper like `get_job_ui_url()` that reuses the same port forward approach already used for Spark Connect in `get_connect_url()`. I think monitoring can be a separate follow up since it is really its own feature (the Prometheus sidecar) and not just a missing wire.

Let me know if this sounds like the right direction before I start on a PR.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.