kestra-io / kestra-io/plugin-spark

Feature Request: Add Resume Functionality Support to Spark Plugin

Open
#115 1 comment 0 reactions 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
5
Forks
5
Avg merge
2d 18h
Merged PRs (30d)
2

Description

### Feature description

## Question: Why doesn't the Spark plugin support resume functionality?

I would like to understand the design decision behind not supporting resume functionality in the Kestra Spark plugin, and request consideration for adding this feature.

## Problem Description

The current Spark plugin does not support job resumption, which causes data consistency issues in production scenarios:

### Scenario 1: Kestra Worker Restart/Deployment

When a Kestra Worker node restarts or deploys a new version while a Spark job is running:
- **Current Behavior**: After Worker restart, Kestra creates a brand new Spark task to retry
- **Problem**: If the user's Spark job uses `INSERT INTO` (append mode) instead of `INSERT OVERWRITE` (overwrite mode), the retry will cause duplicate data writes

## Request

Could you please clarify:
1. **Why resume functionality was not included** in the initial design?
2. **What are the technical or design considerations** that prevent or complicate adding resume support?

## Suggestion

We suggest adding resume functionality support to the Spark plugin. This would allow:
- Resuming Spark jobs when Worker restarts, instead of creating new tasks

## Priority

**High Priority** - This affects data consistency in production environments.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files or tests. Start by reviewing the Spark plugin's job-submission lifecycle and Kestra Worker restart and retry behavior, especially for INSERT INTO jobs. Done would require an agreed resume design and evidence that a worker restart does not create a duplicate Spark submission.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.