apache / apache/auron

Spark SQL job does not terminate after successful execution, continues to spawn new jobs repeatedly

Open
#2,202 0 comments 0 reactions 1 assignee Claimed by @slfan1989 View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

## Description
When executing a simple Spark SQL query that should only submit a single job, the job successfully completes but fails to terminate. Instead, it continues to spawn new jobs in a repetitive cycle without exiting.

## Reproduction Steps
1. Submit a simple Spark SQL query (e.g., `SELECT * FROM table WHERE condition = '432'`)
2. Observe the job execution in the Aurora interface
3. Note that after the initial job succeeds, new jobs are continuously spawned

## Expected Behavior
- The Spark SQL job should terminate normally after successful execution
- No additional jobs should be spawned after the initial job completes

## Actual Behavior
- The job continues to run indefinitely, repeatedly spawning new jobs
- The system does not exit or terminate as expected

## Screenshot/Logs
![Job execution screenshot](https://t1.chatglm.cn/file/69de05354cb0fcbae6871523.png?expired_at=1776590197&sign=9c2110d239689a2d1007166b1443ba39&ext=png)
- Shows job ID 3 with successful job IDs list but running job ID [576] still active
- Multiple successful job IDs indicate repeated execution

## Environment Information
- Apache Aurora version: 7.0.0
- Spark version: 3.2.4
- Hadoop version: 3.0.0

## Additional Context
This issue occurs with simple queries that should only generate a single job, suggesting a potential termination or cleanup problem in the job lifecycle management.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.