apache / apache/seatunnel

[Improve][Connector-V2][Starrocks-Source]When obtaining data from Starrock, an exception prompt 'Unknown HostException' appears

Open
#10,123 8 comments 0 reactions 1 assignee Claimed by @LiJie20190102 View on GitHub
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

When I retrieve data from Starrocks, the configuration is as follows:

```
env {
parallelism = 1
job.mode = "BATCH"

spark.app.name = "SeaTunnel"
spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "1g"
spark.master = local
}

source {
StarRocks {
nodeUrls = ["localhost:32529"]
username = xxxx
password = "xxxx"
database = "db01"
table = "user"
max_retries = 3
schema {
fields {
name1 = STRING
name2 = STRING
name3 = STRING
}
}
}
}

transform {
}

sink {
Console {
parallelism = 1
}
}
```

At this point, an error log will appear:

Image

```
Caused by: org.apache.seatunnel.connectors.seatunnel.starrocks.exception.StarRocksConnectorException: ErrorCode:[STARROCKS-04], ErrorDescription:[Create StarRocks BE reader failed] - Failed to open socket
at org.apache.seatunnel.connectors.seatunnel.starrocks.client.source.StarRocksBeReadClient.(StarRocksBeReadClient.java:88)
at org.apache.seatunnel.connectors.seatunnel.starrocks.source.StarRocksSourceReader.read(StarRocksSourceReader.java:113)
at org.apache.seatunnel.connectors.seatunnel.starrocks.source.StarRocksSourceReader.pollNext(StarRocksSourceReader.java:75)
at org.apache.seatunnel.translation.source.ParallelSource.run(ParallelSource.java:144)
at org.apache.seatunnel.translation.spark.source.partition.batch.ParallelBatchPartitionReader.lambda$prepare$0(ParallelBatchPartitionReader.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
Caused by: com.starrocks.shade.org.apache.thrift.transport.TTransportException: java.net.UnknownHostException: pingt-7f5cf4cfdc-cn-0.pingt-7f5cf4cfdc-cn-headless.olap-c9cd-11f0-aec5-c245473c61c7-61f3bdda.svc.cluster.local
at com.starrocks.shade.org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.seatunnel.connectors.seatunnel.starrocks.client.source.StarRocksBeReadClient.(StarRocksBeReadClient.java:85)
... 12 more
Caused by: java.net.UnknownHostException: pingt-7f5cf4cfdc-cn-0.pingt-7f5cf4cfdc-cn-headless.olap-c9cd-11f0-aec5-c245473c61c7-61f3bdda.svc.cluster.local
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.starrocks.shade.org.apache.thrift.transport.TSocket.open(TSocket.java:221)
... 13 more
```

I noticed that the code obtains the host and port 9060 of BE through the FE address, and returns the domain name. However, our starrocks are deployed through k8s and are not in the same cluster as our Spark cluster, which can cause an exception:

Image

My solution idea is that the host and port 9060 of BE can be obtained through configuration, or if the user has not configured this information, it can be obtained through FE information. What do you think?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.