apache / apache/seatunnel

[Bug] [MySQL-CDC to Doris] 流式任务状态显示失败但是任务仍在运行,也无法手工停止

Open
#8,065 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 17h
Merged PRs (30d)
210

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

使用zeta引擎分离模式部署一个master,一个worker节点,使用mysqlcdc同步数据到doris中,运行一段时间后查看任务状态显示FAILED。于是手工恢复任务,但是一会任务又显示失败,但是查看doris发现仍有数据写入,查看失败任务的日志,发现任务日志仍在在更新。

### SeaTunnel Version

2.3.8

### SeaTunnel Config

```conf
env {
job.name = "stream-test"
job.mode = "STREAMING"
checkpoint.interval = 60000
checkpoint.timeout = 60000
parallelism = 4
read_limit.rows_per_second=100
}

source {
MySQL-CDC {
base-url = "jdbc:mysql://127.0.0.1:3306/testdb"
username = "test_user"
password = "test_pwd"
snapshot.fetch.size = 200
connection.pool.size = 4
startup.mode = "initial"
database-names = [
"testdb"
]
table-names = [
"testdb.t1",
"testdb.t2",
"testdb.t3",
"testdb.t4"
]
debezium {
include.schema.changes = true
}
}
}

sink {
Doris {
fenodes = "127.0.0.1:8030"
query-port = 9030
username = "test_user"
password = "test_pwd"
database = "${database_name}"
table = "${table_name}"
sink.enable-delete = "true"
doris.config {
format = "json"
read_json_by_line = "true"
}
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
save_mode_create_template = """
CREATE TABLE IF NOT EXISTS `${database}`.`${table_name}` (
${rowtype_primary_key},
${rowtype_fields},
sync_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=OLAP
UNIQUE KEY (${rowtype_primary_key})
DISTRIBUTED BY HASH (${rowtype_primary_key})
PROPERTIES (
"replication_allocation" = "tag.location.default: 3",
"in_memory" = "false",
"storage_format" = "V2",
"disable_auto_compaction" = "false"
)
"""
}
}
```

### Running Command

```shell
./bin/seatunnel.sh -c job/stream-test.config -r 909634676551843842 --async
```

### Error Exception

```log
org.apache.seatunnel.engine.common.exception.TaskGroupDeployException: java.lang.RuntimeException: TaskGroupLocation: TaskGroupLocation{jobId=909634676551843842, pipelineId=1, taskGroupId=1} already exists
at org.apache.seatunnel.engine.server.TaskExecutionService.deployTask(TaskExecutionService.java:337)
at org.apache.seatunnel.engine.server.TaskExecutionService.deployTask(TaskExecutionService.java:268)
at org.apache.seatunnel.engine.server.task.operation.DeployTaskOperation.runInternal(DeployTaskOperation.java:55)
at org.apache.seatunnel.engine.server.task.operation.TracingOperation.run(TracingOperation.java:44)
at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)

at org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex.stateProcess(PhysicalVertex.java:574)
at org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex.updateTaskState(PhysicalVertex.java:403)
at org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex.makeTaskGroupDeploy(PhysicalVertex.java:317)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.lambda$stateProcess$19(SubPlan.java:643)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.stateProcess(SubPlan.java:639)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.updatePipelineState(SubPlan.java:383)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.stateProcess(SubPlan.java:633)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.updatePipelineState(SubPlan.java:383)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.stateProcess(SubPlan.java:624)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.startSubPlanStateProcess(SubPlan.java:608)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.restorePipeline(SubPlan.java:496)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.stateProcess(SubPlan.java:676)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.updatePipelineState(SubPlan.java:383)
at org.apache.seatunnel.engine.server.dag.physical.SubPlan.lambda$addPhysicalVertexCallBack$4(SubPlan.java:228)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```

### Zeta or Flink or Spark Version

zeta: 2.3.8

### Java or Scala Version

jdk:1.8.0_111

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the streaming job with the supplied MySQL-CDC-to-Doris configuration and command, then trace TaskExecutionService.java, DeployTaskOperation.java, PhysicalVertex.java, and SubPlan.java from the reported TaskGroupDeployException. Done means the displayed task state matches whether the task is actually running, and manual stopping no longer leaves the task active.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.