alibaba / alibaba/DataX

starrocks-writer在连接Fe节点时,能不能修改默认超时时间为30S,否则单个Fe节点的集群在写数据时容易报:connect time out

Open
#2,229 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

当使用starrocks-writer插件写数据至sr集群的时候,完全没有规律的会报一下异常
: Failed to connect to address:http://x.x.x.x:8030 java.net.SocketTimeOutException: connect time out
会导致整个datax的采集任务失败;
当我查看了下starrocks-writer插件源代码,发现每次写数据至sr集群时,会测试当前fe节点是否正常连接,我把这段代码复制出来进行单独测试,发现平均1K次会有0-5次左右失败,错误和上面的异常是能对应的;最终参考flink-starrocks插件,代码都相同,只是超时时间datax设置的1s,flink设置的是30秒,当把datax的超时时间修改为30s后,生产环境就比较稳定。

在sr集群中fe节点只有1个的时候这种情况比较常见。如果是3个节点那大概率是不会发生的。

所以能不能将datax中设置超时时间和flink中保持一致,都默认设置为30S?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the starrocks-writer plugin source where the FE node connection is checked, and compare its timeout with the referenced flink-starrocks implementation. Set the default connection timeout to 30 seconds and verify that the writer handles single-FE connection attempts without the reported timeout failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.