kestra-io / kestra-io/plugin-spark
Feature Request: Add Resume Functionality Support to Spark Plugin
- Dominant language
- Java
- Stars
- 5
- Forks
- 5
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 2
Description
### Feature description
## Question: Why doesn't the Spark plugin support resume functionality?
I would like to understand the design decision behind not supporting resume functionality in the Kestra Spark plugin, and request consideration for adding this feature.
## Problem Description
The current Spark plugin does not support job resumption, which causes data consistency issues in production scenarios:
### Scenario 1: Kestra Worker Restart/Deployment
When a Kestra Worker node restarts or deploys a new version while a Spark job is running:
- **Current Behavior**: After Worker restart, Kestra creates a brand new Spark task to retry
- **Problem**: If the user's Spark job uses `INSERT INTO` (append mode) instead of `INSERT OVERWRITE` (overwrite mode), the retry will cause duplicate data writes
## Request
Could you please clarify:
1. **Why resume functionality was not included** in the initial design?
2. **What are the technical or design considerations** that prevent or complicate adding resume support?
## Suggestion
We suggest adding resume functionality support to the Spark plugin. This would allow:
- Resuming Spark jobs when Worker restarts, instead of creating new tasks
## Priority
**High Priority** - This affects data consistency in production environments.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files or tests. Start by reviewing the Spark plugin's job-submission lifecycle and Kestra Worker restart and retry behavior, especially for INSERT INTO jobs. Done would require an agreed resume design and evidence that a worker restart does not create a duplicate Spark submission.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100