apache / apache/seatunnel

[Bug] [Zeta] After a checkpoint timeout occurs in the source pipeline, its status changes to CANCELED, but the task remains in a RUNNING state

Open
#9,368 19 comments 1 reaction 0 assignees View on GitHub
bug waiting for review
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

当Source pipeline出现Checkpoint timeout后,Source pipeline的状态会变为CANCELED并且不会恢复,但任务实际一直处于运行状态,且速率为0。

![Image](https://github.com/user-attachments/assets/70873781-0b5f-489c-aab6-d0a477be1ac8)

![Image](https://github.com/user-attachments/assets/7c2077dc-73e8-4349-8532-4e544c7f85c0)

![Image](https://github.com/user-attachments/assets/06f7cfcc-bbfe-40b9-a40d-d9e3f6b479eb)

![Image](https://github.com/user-attachments/assets/b10709a3-f67b-426c-a5f5-d1f5fe1b26d4)

通过bin/seatunnel.sh -j jobId查看任务信息,可以看到 pipelineStateMapperMap 的状态为 CANCELING,而内部的 executionStateMap 中的两个 TaskGroupLocaltion 对象中一个为 CANCELED,另一个一直卡在 CANCELING

![Image](https://github.com/user-attachments/assets/6c440268-0b11-4cb1-8f95-f2c2da59be7c)

### SeaTunnel Version

2.3.10

### SeaTunnel Config

```conf
{
"env": {
"checkpoint.interval": 300000,
"checkpoint.timeout": 10000,
"job.mode": "BATCH",
"parallelism": 1
},
"sink": [
{
"data_save_mode": "APPEND_DATA",
"database": "c_test",
"doris.batch.size": 4096,
"doris.config": {
"column_separator": "\t",
"enclose": "#",
"format": "csv",
"trim_double_quotes": true
},
"fenodes": "******:8030",
"password": "******",
"plugin_name": "DORIS",
"sink.buffer-count": 3,
"sink.buffer-size": "256 * 1024",
"sink.enable-2pc": false,
"sink.label-prefix": "custom_fields-1747459814",
"sink.max-retries": 3,
"table": "custom_fields",
"username": "******"
}
],
"source": [
{
"access_key": "******",
"access_secret": "******",
"bucket": "obs://******",
"endpoint": "https://******",
"field_delimiter": "\u0001",
"file_format_type": "text",
"path": "/data/20250527/174000/1748338893139",
"plugin_name": "OBSFILE",
"schema": {
"fields": {
"ATTRIBUTE": "STRING",
"CREATED_DATE_TIME": "TIMESTAMP",
"HADP_RS_CUSTOM_FIELDS_TRANSACTION_ID": "BIGINT",
"HADP_RS_CUSTOM_FIELDS_TRANS_NUMBER": "BIGINT",
"HADP_RS_CUSTOM_FIELDS_TRANS_TYPE": "SHORT",
"HANDLE": "STRING",
"MODIFIED_DATE_TIME": "TIMESTAMP",
"VALUE": "STRING"
}
},
"skip_header_row_number": 1,
"tmp_path": "/tmp"
}
],
"transform": [
{
"plugin_name": "SQL",
"query": "select ATTRIBUTE, CREATED_DATE_TIME, HANDLE, MODIFIED_DATE_TIME, VALUE Duplicate of #/from dual"
}
}
```

### Running Command

```shell
bin/seatunnel.sh -c doris.json
```

### Error Exception

```log
[979681127113424900] 2025-05-27 17:48:33,061 INFO [.s.e.s.c.CheckpointCoordinator] [checkpoint-coordinator-1/979681127113424900] - timeout checkpoint:
[979681127113424900]/1/1, CHECKPOINT_TYPE
[979681127113424900] 2025-05-27 17:48:33,066 INFO [.s.e.s.c.CheckpointCoordinator] [checkpoint-coordinator-1/979681127113424900] - start clean pending checkpoint cause Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml or jobConfig env.
[979681127113424900] 2025-05-27 17:48:33,066 ERROR [s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-749] - trigger checkpoint failed org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml or jobConfig env.
at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.abortCheckpoint(PendingCheckpoint.java:176) ~[seatunnel-starter.jar:2.3.10]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$cleanPendingCheckpoint$20(CheckpointCoordinator.java:789) ~[seatunnel-starter.jar:2.3.10]
at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4705) ~[?:1.8.0_362]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.cleanPendingCheckpoint(CheckpointCoordinator.java:787) ~[seatunnel-starter.jar:2.3.10]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:288) ~[seatunnel-starter.jar:2.3.10]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$startTriggerPendingCheckpoint$9(CheckpointCoordinator.java:667) ~[seatunnel-starter.jar:2.3.10]
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43) ~[seatunnel-starter.jar:2.3.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_362]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_362]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
```

### Zeta or Flink or Spark Version

Zeta

### Java or Scala Version

_No response_

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with CheckpointCoordinator.cleanPendingCheckpoint and PendingCheckpoint.abortCheckpoint, which appear in the reported stack trace, then reproduce the timeout with the supplied bin/seatunnel.sh command and configuration. Trace how pipelineStateMapperMap and executionStateMap are updated after the checkpoint failure; done means the source pipeline and its task groups reach a consistent terminal state instead of remaining CANCELING or RUNNING.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.