[Bug][Zeta] checkpoint-coordinator thread survives job cancellation (CANCELED job leaves its coordinator executor alive)
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 203
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue) and found no similar issues.
### What happened
After a streaming job is cancelled and reaches `CANCELED`, the Zeta node still has the job's checkpoint coordinator thread alive. The e2e framework's post-job thread check (`SeaTunnelContainer` "There are still threads running in the container") catches it, and the two file-source continuous-discovery tests that cancel their job at the end fail with `Wait continuous job exit failed`:
- PR #11757, fork run `waterWang/seatunnel` 33998012457, job `connector-file-sftp-it (11)` (job 101392143762): `SftpFileIT#testSftpBinaryUpdateModeContinuousDiscoveryPostSyncBackup`, Zeta leg.
- PR #11727, fork run `abdessalems/seatunnel` 34022796412, job `all-connectors-it-5 (11)` (job 101458676526): `FtpFileIT#testFtpBinaryUpdateModeContinuousDiscoveryWithNonRecursiveScan`, Zeta leg.
Both PRs touch neither the file connectors nor the checkpoint coordinator.
### Evidence
Sequence in both runs: `container.cancelJob(jobId)` returns 0, `getJobStatus(jobId)` becomes `CANCELED` within the 30 s wait, the engine logs `Job turned from state CANCELING to CANCELED` and `Job () end with state CANCELED`, and then the thread check that `executeJob` runs after the job keeps reporting for 2 minutes:
```
There are still threads running in the container:
"checkpoint-coordinator-1/5309008626678301139" #317 prio=5 os_prio=0 tid=... waiting on condition
java.lang.Thread.State: WAITING (parking)
- parking to wait for <0x00000000f58ff6d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
```
(the FTP run; the SFTP run shows the same single `checkpoint-coordinator-1/` thread). The thread name is the per-job checkpoint coordinator executor of the JobMaster; it is parked on its queue, i.e. the executor was never shut down when the pipeline ended in `CANCELED`.
### What you expected to happen
Cancelling a job must release the job's checkpoint coordinator executor together with the rest of the JobMaster resources, so no `checkpoint-coordinator-*` thread outlives a job that has reached a terminal state. The e2e thread-leak check exists precisely to catch this class of leak on the Zeta node.
### SeaTunnel Version
dev (`96836fc6`), Zeta.
### Engine
Zeta
### Additional context
Only seen on the two continuous-discovery tests so far because they are among the few e2e cases that cancel a streaming job and then run the thread check; batch jobs end via `FINISHED` and do not show it. Filed while triaging CI for #11757 and #11727.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Zeta JobMaster cancellation and resource-shutdown path, then inspect the checkpoint coordinator executor named in the report. Reproduce the leak with SftpFileIT#testSftpBinaryUpdateModeContinuousDiscoveryPostSyncBackup or FtpFileIT#testFtpBinaryUpdateModeContinuousDiscoveryWithNonRecursiveScan and follow SeaTunnelContainer's post-job thread check. Done means a canceled job reaches CANCELED without any checkpoint-coordinator thread remaining.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100