apache / apache/seatunnel

[Bug] [PostgresSQL] pg-doris流式任务中断

Open
#9,852 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

我们从PostgreSQL到Doris的SeaTunnel流作业,每天凌晨0-1点左右都会因doris连接超时错误而失败。这项工作在一天的大部分时间里都能成功运行,但在清晨总是失败。我们doris库是新建的也不存在性能问题,报错的时间点也检查了doris环境以及各个be/fe节点均未发现异常。

### SeaTunnel Version

2.3.9

### SeaTunnel Config

```conf
"sink": [{
"fenodes": #{url},
"query-port": #{queryPort},
"username": #{user},
"password": #{password},
"database": #{database},
"table": "${table_name}",
"source_table_name" : #{source_table_name},
"sink.enable-2pc": "false",
"doris.batch.size": 10000,
"sink.buffer-size": 10485760,
"sink.check-interval": 50000,
"sink.max-retries": 5,
"sink.enable-delete":true,
"plugin_name" : "Doris",
"schema_save_mode":"ERROR_WHEN_SCHEMA_NOT_EXIST",
"doris.config": {
"format": "json",
"read_json_by_line": "true",
"num_as_string":"true"
}
}]
```

### Running Command

```shell
seatunnel.sh
```

### Error Exception

```log
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: ErrorCode:[Doris-01], ErrorDescription:[stream load error]
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:302) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:70) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:70) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:50) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:51) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:72) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:169) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:77) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:694) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1019) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43) ~[seatunnel-starter.jar:2.3.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_451]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_451]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_451]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_451]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_451]
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: ErrorCode:[Doris-01], ErrorDescription:[stream load error]
at org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter.prepareCommit(MultiTableSinkWriter.java:276) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:201) ~[seatunnel-starter.jar:2.3.9]
... 17 more
Caused by: java.util.concurrent.ExecutionException: org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: ErrorCode:[Doris-01], ErrorDescription:[stream load error]
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_451]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_451]
at org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter.prepareCommit(MultiTableSinkWriter.java:274) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:201) ~[seatunnel-starter.jar:2.3.9]
... 17 more
Caused by: org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: ErrorCode:[Doris-01], ErrorDescription:[stream load error]
at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.stopLoad(DorisStreamLoad.java:228) ~[?:?]
at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.flush(DorisSinkWriter.java:221) ~[?:?]
at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.prepareCommit(DorisSinkWriter.java:206) ~[?:?]
at org.apache.seatunnel.api.sink.SinkWriter.prepareCommit(SinkWriter.java:75) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter.lambda$prepareCommit$4(MultiTableSinkWriter.java:259) ~[seatunnel-starter.jar:2.3.9]
... 6 more
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: 连接超时 (Read failed)
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_451]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_451]
at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.stopLoad(DorisStreamLoad.java:226) ~[?:?]
at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.flush(DorisSinkWriter.java:221) ~[?:?]
at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.prepareCommit(DorisSinkWriter.java:206) ~[?:?]
at org.apache.seatunnel.api.sink.SinkWriter.prepareCommit(SinkWriter.java:75) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter.lambda$prepareCommit$4(MultiTableSinkWriter.java:259) ~[seatunnel-starter.jar:2.3.9]
... 6 more
```

### Zeta or Flink or Spark Version

_No response_

### Java or Scala Version

_No response_

### Screenshots

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with connectors/doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/DorisStreamLoad.java and DorisSinkWriter.java, then trace the failure through MultiTableSinkWriter.prepareCommit. Reproduce the scheduled PostgreSQL-to-Doris stream failure around midnight and collect the complete Doris error details. Done means the reported timeout is reproducible and its handling or cause is addressed with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
data-engineering, databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.