apache / apache/hudi

[SUPPORT]The k8s cluster submitted a task to write Spark streaming to Hudi, but encountered an error

Open
#10,699 2 comments 0 reactions 0 assignees View on GitHub
component:timeline-server engine:spark priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Caused by: java.net.UnknownHostException: mytask-driver-svc.dataaccess.svc: Temporary failure in name resolution
at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(Unknown Source)
at java.base/java.net.InetAddress.getAddressesFromNameService(Unknown Source)
at java.base/java.net.InetAddress$NameServiceAddresses.get(Unknown Source)
at java.base/java.net.InetAddress.getAllByName0(Unknown Source)
at java.base/java.net.InetAddress.getAllByName(Unknown Source)
at java.base/java.net.InetAddress.getAllByName(Unknown Source)
at org.apache.hudi.org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.hudi.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111)
at org.apache.hudi.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.hudi.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.hudi.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.hudi.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.hudi.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.hudi.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.hudi.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.hudi.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.hudi.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.hudi.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.hudi.org.apache.http.client.fluent.Request.execute(Request.java:151)
at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.executeRequestToTimelineServer(TimelineServerBasedWriteMarkers.java:173)
at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.doesMarkerDirExist(TimelineServerBasedWriteMarkers.java:99)
... 57 more

I found this Serivce 'mytask-driver-svc.dataaccess.svc' in my k8s cluster.This is an occasional problem

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported Kubernetes service name, DNS error, and the Hudi TimelineServerBasedWriteMarkers stack-trace entries. Check the cluster’s service and name-resolution behavior around the intermittent failure; done means identifying a reproducible cause and documenting a confirmed resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes, spark
Domain
data-engineering, distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.