apache / apache/seatunnel

[Bug][Zeta] checkpoint-coordinator thread survives job cancellation (CANCELED job leaves its coordinator executor alive)

Open
#12,147 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue) and found no similar issues.

### What happened

After a streaming job is cancelled and reaches `CANCELED`, the Zeta node still has the job's checkpoint coordinator thread alive. The e2e framework's post-job thread check (`SeaTunnelContainer` "There are still threads running in the container") catches it, and the two file-source continuous-discovery tests that cancel their job at the end fail with `Wait continuous job exit failed`:

- PR #11757, fork run `waterWang/seatunnel` 33998012457, job `connector-file-sftp-it (11)` (job 101392143762): `SftpFileIT#testSftpBinaryUpdateModeContinuousDiscoveryPostSyncBackup`, Zeta leg.
- PR #11727, fork run `abdessalems/seatunnel` 34022796412, job `all-connectors-it-5 (11)` (job 101458676526): `FtpFileIT#testFtpBinaryUpdateModeContinuousDiscoveryWithNonRecursiveScan`, Zeta leg.

Both PRs touch neither the file connectors nor the checkpoint coordinator.

### Evidence

Sequence in both runs: `container.cancelJob(jobId)` returns 0, `getJobStatus(jobId)` becomes `CANCELED` within the 30 s wait, the engine logs `Job turned from state CANCELING to CANCELED` and `Job () end with state CANCELED`, and then the thread check that `executeJob` runs after the job keeps reporting for 2 minutes:

```
There are still threads running in the container:
"checkpoint-coordinator-1/5309008626678301139" #317 prio=5 os_prio=0 tid=... waiting on condition
java.lang.Thread.State: WAITING (parking)
- parking to wait for <0x00000000f58ff6d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
```

(the FTP run; the SFTP run shows the same single `checkpoint-coordinator-1/` thread). The thread name is the per-job checkpoint coordinator executor of the JobMaster; it is parked on its queue, i.e. the executor was never shut down when the pipeline ended in `CANCELED`.

### What you expected to happen

Cancelling a job must release the job's checkpoint coordinator executor together with the rest of the JobMaster resources, so no `checkpoint-coordinator-*` thread outlives a job that has reached a terminal state. The e2e thread-leak check exists precisely to catch this class of leak on the Zeta node.

### SeaTunnel Version

dev (`96836fc6`), Zeta.

### Engine

Zeta

### Additional context

Only seen on the two continuous-discovery tests so far because they are among the few e2e cases that cancel a streaming job and then run the thread check; batch jobs end via `FINISHED` and do not show it. Filed while triaging CI for #11757 and #11727.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Zeta JobMaster cancellation and resource-shutdown path, then inspect the checkpoint coordinator executor named in the report. Reproduce the leak with SftpFileIT#testSftpBinaryUpdateModeContinuousDiscoveryPostSyncBackup or FtpFileIT#testFtpBinaryUpdateModeContinuousDiscoveryWithNonRecursiveScan and follow SeaTunnelContainer's post-job thread check. Done means a canceled job reaches CANCELED without any checkpoint-coordinator thread remaining.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.