kubeflow / kubeflow/spark-operator

Detect Pending_rerun state for spark application as a prometheus metric

Open
#1,752 8 comments 1 reaction 0 assignees View on GitHub
lifecycle/frozen
Dominant language
Python
Stars
3.2k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
13

Description

I am using a
apiVersion: "sparkoperator.k8s.io/v1beta2"
I have a kind
**kind: SparkApplication**
under which there are many spark jobs. However one of the **SparkApplication** is under PENDING_RERUN state, this is now for more than 35 days . We want to detect this problem and give a prometheus alert. Is there any prometheus query that catches this condition?
my kubernetes query gives this result.

**kubectl get sparkapplication -n <>
NAME STATUS ATTEMPTS START FINISH AGE
A PENDING_RERUN 35d**

I had a look at documentation and used all of these

"spark_app_count"
"spark_app_submit_count"
"spark_app_success_count"
"spark_app_failure_count"
"spark_app_running_count"

however none of these catch the PENDING_RERUN state.

thanks in advance

Contributor guide

Open the contributing guide

Research direction

Start with the SparkApplication metrics documentation and the existing spark_app_count, spark_app_submit_count, spark_app_success_count, spark_app_failure_count, and spark_app_running_count metrics. Determine how the PENDING_RERUN state is represented, then define a Prometheus metric or query that detects it and verify that it supports an alert for applications remaining in that state.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, prometheus, spark
Domain
devops, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.