apache / apache/airflow

Consider adding deferrable support to `SparkSubmitOperator`

Open
#67,168 2 comments 1 reaction 1 assignee Claimed by @amoghrajesh View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

## Context

PR #67118 is attempting to add sync resumable support to `SparkSubmitOperator` via `ResumableJobMixin`. During review, @ashb suggested also supporting a `deferrable=True` mode in the same operator.

## What this issue tracks

Add `deferrable: bool = False` parameter to `SparkSubmitOperator`:

- `deferrable=False` (default): sync path with `ResumableJobMixin` — worker slot occupied during polling, reconnects to existing driver on infrastructure failure (implemented in #67118)
- `deferrable=True`: submit job, `defer()` to `SparkDriverTrigger`, worker slot freed during polling. When `execute()` is called again (only happens on user clear), resubmit fresh — no reconnect needed since crashes are handled by Trigger row persistence.

Relationship to #67118: The two modes share `spark_job_id` in `task_state`. A user can switch from sync to deferrable without any state migration.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.