apache / apache/seatunnel

[Bug] [connector-sftp] Concurrent write sftp error

Open
#9,870 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

I run `run.sh 20`. My source MySQL database has approximately 3.8 million records

### SeaTunnel Version

dev

### SeaTunnel Config

```conf
env {
"job.mode"="BATCH"
"job.name"="task_12_20240308"
}
source {
Jdbc {
url="jdbc:mysql://127.0.0.1:3306/test?useUnicode=true&rewriteBatchedStatements=true&serverTimezone=Asia%2FShanghai&zeroDateTimeBehavior=convertToNull&characterEncoding=utf-8&useSSL=false"
driver="com.mysql.cj.jdbc.Driver"
schema="test"
user="root"
password="xxx"
query="SELECT `pl_curr_cd`,`murex_intrnl_tran_id`,`murex_port_cd`,`murex_dts_ref`,`batch_dt` FROM `bj`.`ds_lv1_pl_details_test` WHERE 1=1 "
"fetch_size"="1000"
"result_table_name"="E000001_source_1_12"
"parallelism"=1
"empty_data_strategy"=true
"table_path"=1
}
}
transform {
}
sink {
LocalFile {
path="/dsg/20240308"
source_table_name="E000001_source_1_12"
tmp_path="/tmp/seatunnel"
custom_filename=true
file_name_expression="ds_lv1_pl_details_test.txt_${transactionId}"
is_enable_transaction=false
filename_time_format="yyyy.MM.dd"
file_format_type="text"
field_delimiter="\u0001"
row_delimiter="\n"
sink_columns=["pl_curr_cd","murex_intrnl_tran_id","murex_port_cd","murex_dts_ref","batch_dt"]
batch_size="1000000"
compress_codec="none"
date_format="yyyy-MM-dd"
datetime_format="yyyy-MM-dd HH:mm:ss"
time_format="HH:mm:ss"
encoding="UTF-8"
}
}
```

### Running Command

```shell
CONCURRENCY=1 # 并发数量,根据服务器性能调整
SEATUNNEL_SCRIPT="../bin/seatunnel.sh"
TEMP_DIR="./temp_configs" # 临时配置文件目录

# 创建临时目录存放配置文件
mkdir -p "$TEMP_DIR"

# 生成随机日期的函数(格式:YYYYMMDD)
generate_random_date() {
local year=$((2020 + RANDOM % 5)) # 2020-2024年
local month=$((1 + RANDOM % 12))
local day=$((1 + RANDOM % 28)) # 避免处理2月29日等复杂情况

# 格式化月份和日期为两位数
printf "%d%02d%02d" "$year" "$month" "$day"
}

# 生成并执行任务的函数
execute_task() {
local task_id=$1
local random_date=$(generate_random_date)

# 生成配置文件内容
local config_content=$(cat << EOF
env {
"job.mode"="BATCH"
"job.name"="task_${task_id}_${random_date}"
}
source {
Jdbc {
url="jdbc:mysql://127.0.0.1:3306/test?useUnicode=true&rewriteBatchedStatements=true&serverTimezone=Asia%2FShanghai&zeroDateTimeBehavior=convertToNull&characterEncoding=utf-8&useSSL=false"
driver="com.mysql.cj.jdbc.Driver"
schema="test"
user="root"
password="xxx"
query="SELECT \`pl_curr_cd\`,\`murex_intrnl_tran_id\`,\`murex_port_cd\`,\`murex_dts_ref\`,\`batch_dt\` FROM \`bj\`.\`ds_lv1_pl_details_test\` WHERE 1=1 "
"fetch_size"="1000"
"result_table_name"="E000001_source_1_${task_id}"
"parallelism"=1
"empty_data_strategy"=true
"table_path"=1
}
}
transform {
}
sink {
SftpFile {
host="172.20.100.238"
port="22"
user="root"
password="xxx"
path="/test/${random_date}"
source_table_name="E000001_source_1"
tmp_path="/tmp/seatunnel"
custom_filename=true
file_name_expression="ds_lv1_pl_details_test.txt_\${transactionId}"
is_enable_transaction=false
filename_time_format="yyyy.MM.dd"
file_format_type="text"
field_delimiter="\u0001"
row_delimiter="\n"
sink_columns=["pl_curr_cd","murex_intrnl_tran_id","murex_port_cd","murex_dts_ref","batch_dt"]
batch_size="1000000"
compress_codec="none"
date_format="yyyy-MM-dd"
datetime_format="yyyy-MM-dd HH:mm:ss"
time_format="HH:mm:ss"
encoding="UTF-8"
}
}
EOF
)

# 写入临时配置文件
local config_file="${TEMP_DIR}/task_${task_id}_${random_date}.conf"
echo "$config_content" > "$config_file"

echo "$(date '+%Y-%m-%d %H:%M:%S') - 开始执行任务 $task_id,随机日期: $random_date"

# 执行 Seatunnel 任务
"$SEATUNNEL_SCRIPT" -c "$config_file"

local exit_code=$?

if [ $exit_code -eq 0 ]; then
echo "$(date '+%Y-%m-%d %H:%M:%S') - 任务 $task_id (日期: $random_date) 执行成功"
else
echo "$(date '+%Y-%m-%d %H:%M:%S') - 任务 $task_id (日期: $random_date) 执行失败,退出码: $exit_code"
fi

return $exit_code
}

# 主执行逻辑
main() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - 开始并发执行任务,并发数: $CONCURRENCY"
# 存储进程ID和日期的数组
declare -a pids
declare -a task_dates
declare -a task_ids

# 启动并发任务
for ((i=1; i<=CONCURRENCY; i++)); do
execute_task $i &
pids[$i]=$!
task_ids[$i]=$i
done

# 等待所有任务完成
for ((i=1; i<=CONCURRENCY; i++)); do
wait ${pids[$i]}
exit_codes[$i]=$?
done
# 输出执行结果摘要
echo ""
echo "======= 任务执行结果摘要 ======="
for ((i=1; i<=CONCURRENCY; i++)); do
if [ ${exit_codes[$i]} -eq 0 ]; then
echo "任务 ${task_ids[$i]}: 成功"
else
echo "任务 ${task_ids[$i]}: 失败 (退出码: ${exit_codes[$i]})"
fi
done

echo "$(date '+%Y-%m-%d %H:%M:%S') - 所有任务执行完成"
}
# 带参数版本的函数
main_with_args() {
local custom_concurrency=$1

if [ -n "$custom_concurrency" ] && [[ "$custom_concurrency" =~ ^[0-9]+$ ]]; then
CONCURRENCY="$custom_concurrency"
fi

main
}
# 清理临时文件函数
cleanup() {
echo "清理临时配置文件..."
}
# 设置退出时清理
trap cleanup EXIT
# 执行主函数
if [ $# -ge 1 ]; then
main_with_args "$1"
else
main
fi
```

### Error Exception

```log
2025-09-17 13:35:15,739 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Fatal Error,

2025-09-17 13:35:15,739 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2025-09-17 13:35:15,739 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Reason:SeaTunnel job executed failed

2025-09-17 13:35:15,740 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:282)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:278)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:398)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:174)
at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.runInternal(CheckpointErrorReportOperation.java:48)
at org.apache.seatunnel.engine.server.task.operation.TracingOperation.run(TracingOperation.java:42)
at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Aggregate commit error.
at org.apache.seatunnel.engine.server.task.SinkAggregatedCommitterTask.restoreState(SinkAggregatedCommitterTask.java:278)
at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$null$0(NotifyTaskRestoreOperation.java:107)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

... 12 more

at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:220)
... 2 more

2025-09-17 13:35:15,740 ERROR [o.a.s.c.s.SeaTunnel ] [main] -
===============================================================================

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:282)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:278)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointManager.java:398)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:174)
at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.runInternal(CheckpointErrorReportOperation.java:48)
at org.apache.seatunnel.engine.server.task.operation.TracingOperation.run(TracingOperation.java:42)
at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Aggregate commit error.
at org.apache.seatunnel.engine.server.task.SinkAggregatedCommitterTask.restoreState(SinkAggregatedCommitterTask.java:278)
at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$null$0(NotifyTaskRestoreOperation.java:107)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

... 12 more

at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:220)
... 2 more
```

### Zeta or Flink or Spark Version

_No response_

### Java or Scala Version

_No response_

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the concurrent `SftpFile` job with `run.sh 20` and the supplied configuration, then inspect `CheckpointCoordinator.java` and `SinkAggregatedCommitterTask.java:278` around the aggregate commit failure. Compare the SFTP sink behavior with the checkpoint stack trace; done means the concurrent run completes without the reported checkpoint or SFTP commit error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql, shell
Domain
backend, data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.