kubeflow / kubeflow/spark-operator
[Bug]: ScheduledSparkApplication permanently stuck in FailedValidation after fixing invalid spec
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 1.5k
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 13
Description
### What happened?
- [x] ✋ I have searched the open/closed issues and my issue is not listed.
When creating or updating a `ScheduledSparkApplication` with an invalid cron schedule (`spec.schedule`) or an invalid timezone (`spec.timeZone`), the controller detects the parse failure and marks `status.scheduleState` as `FailedValidation` with the corresponding error message in `status.reason`.
However, once a user corrects the specification (e.g., fixing the typo in `spec.schedule` or `spec.timeZone`), the controller validates the new schedule successfully, but the state machine in `Reconcile` hits `case v1beta2.ScheduleStateFailedValidation:` which unconditionally executes `return ctrl.Result{}, nil`.
As a result, the controller never transitions the resource back to `Scheduled`, never calculates `status.nextRun`, and never enqueues a timer. The `ScheduledSparkApplication` becomes permanently deadlocked in `FailedValidation` and will never run again unless it is completely deleted and recreated.
### Reproduction Code
1. Create a `ScheduledSparkApplication` with an invalid cron schedule (e.g. `spec.schedule: "invalid-cron"`):
```yaml
apiVersion: sparkoperator.k8s.io/v1beta2
kind: ScheduledSparkApplication
metadata:
name: test-scheduled-app
namespace: default
spec:
schedule: "invalid-cron"
concurrencyPolicy: Allow
template:
type: Scala
mode: cluster
image: spark:3.5.0
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
restartPolicy:
type: Never
```
2. Inspect the resource status:
```bash
kubectl get scheduledsparkapplication test-scheduled-app -o jsonpath='{.status.scheduleState}'
# Output: FailedValidation
kubectl get scheduledsparkapplication test-scheduled-app -o jsonpath='{.status.reason}'
# Output: expected exactly 5 fields, found 1: [invalid-cron]
```
3. Update the resource with a valid cron expression:
```bash
kubectl patch scheduledsparkapplication test-scheduled-app --type merge -p '{"spec":{"schedule":"@every 5m"}}'
```
4. Inspect the resource status again:
```bash
kubectl get scheduledsparkapplication test-scheduled-app -o yaml
```
### Expected behavior
When `spec.schedule` or `spec.timeZone` is updated to a valid value:
1. `status.scheduleState` should transition to `Scheduled`.
2. `status.reason` should be cleared.
3. `status.nextRun` should be populated with the next scheduled execution time.
4. The reconciler should requeue for `nextRun.Sub(now)`.
### Actual behavior
1. `status.scheduleState` remains permanently stuck in `FailedValidation`.
2. `status.reason` remains set to the old error message.
3. `status.nextRun` is not scheduled.
4. The reconciler returns `ctrl.Result{}, nil`, ignoring all future executions.
### Environment & Versions
- Kubernetes Version: v1.30+
- Spark Operator Version: master / v2.x (controller-runtime architecture)
- Apache Spark Version: N/A
### Additional context
_No response_
### Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
Contributor guide
Research direction
Start at the ScheduledSparkApplication Reconcile path and inspect the FailedValidation state handling described in the issue, along with the schedule and timezone validation flow. Reproduce the failure with an invalid value, correct it using the shown kubectl patch, and verify that the state and reason recover, nextRun is populated, and reconciliation queues the next execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100