kubeflow / kubeflow/spark-operator
Detect Pending_rerun state for spark application as a prometheus metric
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 1.5k
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 13
Description
I am using a
apiVersion: "sparkoperator.k8s.io/v1beta2"
I have a kind
**kind: SparkApplication**
under which there are many spark jobs. However one of the **SparkApplication** is under PENDING_RERUN state, this is now for more than 35 days . We want to detect this problem and give a prometheus alert. Is there any prometheus query that catches this condition?
my kubernetes query gives this result.
**kubectl get sparkapplication -n <>
NAME STATUS ATTEMPTS START FINISH AGE
A PENDING_RERUN 35d**
I had a look at documentation and used all of these
"spark_app_count"
"spark_app_submit_count"
"spark_app_success_count"
"spark_app_failure_count"
"spark_app_running_count"
however none of these catch the PENDING_RERUN state.
thanks in advance
Contributor guide
Research direction
Start with the SparkApplication metrics documentation and the existing spark_app_count, spark_app_submit_count, spark_app_success_count, spark_app_failure_count, and spark_app_running_count metrics. Determine how the PENDING_RERUN state is represented, then define a Prometheus metric or query that detects it and verify that it supports an alert for applications remaining in that state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, prometheus, spark
- Domain
- devops, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100