dotnet / dotnet/spark

.NET for Apache Spark JDBC to connect to Postgresql

Open
#303 7 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

I am new on .NET for Apache Spark. I try connect postgresql db. But get these error:JVM method execution failed: Nonstatic method jdbc failed for class 6 when called with 3 arguments

JVM method ex
ecution failed: Nonstatic method load failed for class 6 when called with no arg
uments

> DOTNET_WORKER_DIR = C:\bin\Microsoft.Spark.Worker-0.5.0
HADOOP_HOME = C:\bin\spark-2.4.1-bin-hadoop2.7\
SPARK_HOME = C:\bin\spark-2.4.1-bin-hadoop2.7\
JAVA_HOME = C:\Program Files\Java\jdk1.8.0_221
Dotnet framework version = 2.2
Nuget spark package = "Microsoft.Spark/0.5.0"

> %SPARK_HOME%\bin\spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local
bin\Debug\netcoreapp2.2\microsoft-spark-2.4.x-0.5.0.jar dotnet bin\Debug\netcoreapp2.2\myapp.dll

````C#
var spark = SparkSession
.Builder().AppName("test_postgresql").GetOrCreate();

DataFrame frame= spark.Read().Format("jdbc").Option("url", "jdbc:postgresql://host:5432/tablename?user=postgres&password=xxxx")
.Option("query", "SELECT name FROM public.\"xxxx\"")
.Load();

Dictionary options = new Dictionary();
options.Add("ssl", "false");
DataFrame dataFrame2 = spark.Read().Jdbc("jdbc:postgresql://host:port/table?user=postgres&password=xxxx", "Roles", options);
````
I tried to type of method to connect postgresql db. İs it possible to connect postgresql db ?

Thanks.

When I try to fix problem, I noticed error log about no suitable driver. So I tried to download Microsoft SQL Server JDBC Driver 2.0 and postgresql jdbc driver and put these jar files under to the C:\bin\spark-2.4.1-bin-hadoop2.7\jars folder.
- postgresql-42.2.8.jar
- sqljdbc.jar
- sqljdbc4.jar

And find example of how to connect to postgresql

````c#
DataFrame jdbcDf = spark.Read()
.Format("jdbc")
.Options(
new Dictionary
{
{"url", "jdbc:postgresql:192.168.10.198:5432/dbname"},
{"dbtable", "Roles"},
{"user", "postgres"},
{"password", "XXXX"},
{"ssl", "false"}
})
.Load();
````
Now I fix "no suitable driver" error with putting jar files but now I take this error :

> [Error] [JvmBridge] org.postgresql.ut
il.PSQLException: Connection to localhost:5432 refused. Check that the hostname
and port are correct and that the postmaster is accepting TCP/IP connections.

But I am not trying to connect localhost .

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the shown SparkSession JDBC calls with the listed .NET for Apache Spark, Spark, PostgreSQL, and JDBC versions. Inspect the JDBC driver and connection error, especially the URL and the reported localhost target. Done means documenting whether PostgreSQL connectivity works with the stated configuration or identifying the configuration issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, postgresql, spark
Domain
data, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.