dotnet / dotnet/spark

Connection refused issue

Open
#1,183 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

Hi, I am a beginner working with spark and dotnet. Let me explain my setup first.
I have a spark master worker setup deployed using bitnami helm chart. Image is custom made to include deltalake and I have a deltatable created and stored in the azure datalake. My requirement was to create an api to take some argument and query the deltatable in the azure datalake. Inorder to do the processing I need spark right?. So I thought I could call the spark in my AKS, pass the arguments to the spark, spark will query it from deltatable located in the azure and return back me the output. I created the api but upon running i am getting the below error

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (111): Connection refused 127.0.0.1:5567

I tried the master url with an external ip, spark headless service, even local port. I presume the port and host is of the visual studio debug configuration. But how can I configure the session builder in the code? Is there any alternative solution to my requirement? Please share some insights , I believe spark-kubernetes-spark.net is a somewhat popular setup, hoping some could help.
Thanks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.