Consider adding deferrable support to `SparkSubmitOperator`
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
## Context
PR #67118 is attempting to add sync resumable support to `SparkSubmitOperator` via `ResumableJobMixin`. During review, @ashb suggested also supporting a `deferrable=True` mode in the same operator.
## What this issue tracks
Add `deferrable: bool = False` parameter to `SparkSubmitOperator`:
- `deferrable=False` (default): sync path with `ResumableJobMixin` — worker slot occupied during polling, reconnects to existing driver on infrastructure failure (implemented in #67118)
- `deferrable=True`: submit job, `defer()` to `SparkDriverTrigger`, worker slot freed during polling. When `execute()` is called again (only happens on user clear), resubmit fresh — no reconnect needed since crashes are handled by Trigger row persistence.
Relationship to #67118: The two modes share `spark_job_id` in `task_state`. A user can switch from sync to deferrable without any state migration.
Contributor guide
Assessment
This issue has not been assessed yet.