.NET for Apache Spark JDBC to connect to Postgresql
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
I am new on .NET for Apache Spark. I try connect postgresql db. But get these error:JVM method execution failed: Nonstatic method jdbc failed for class 6 when called with 3 arguments
JVM method ex
ecution failed: Nonstatic method load failed for class 6 when called with no arg
uments
> DOTNET_WORKER_DIR = C:\bin\Microsoft.Spark.Worker-0.5.0
HADOOP_HOME = C:\bin\spark-2.4.1-bin-hadoop2.7\
SPARK_HOME = C:\bin\spark-2.4.1-bin-hadoop2.7\
JAVA_HOME = C:\Program Files\Java\jdk1.8.0_221
Dotnet framework version = 2.2
Nuget spark package = "Microsoft.Spark/0.5.0"
> %SPARK_HOME%\bin\spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local
bin\Debug\netcoreapp2.2\microsoft-spark-2.4.x-0.5.0.jar dotnet bin\Debug\netcoreapp2.2\myapp.dll
````C#
var spark = SparkSession
.Builder().AppName("test_postgresql").GetOrCreate();
DataFrame frame= spark.Read().Format("jdbc").Option("url", "jdbc:postgresql://host:5432/tablename?user=postgres&password=xxxx")
.Option("query", "SELECT name FROM public.\"xxxx\"")
.Load();
Dictionary options = new Dictionary();
options.Add("ssl", "false");
DataFrame dataFrame2 = spark.Read().Jdbc("jdbc:postgresql://host:port/table?user=postgres&password=xxxx", "Roles", options);
````
I tried to type of method to connect postgresql db. İs it possible to connect postgresql db ?
Thanks.
When I try to fix problem, I noticed error log about no suitable driver. So I tried to download Microsoft SQL Server JDBC Driver 2.0 and postgresql jdbc driver and put these jar files under to the C:\bin\spark-2.4.1-bin-hadoop2.7\jars folder.
- postgresql-42.2.8.jar
- sqljdbc.jar
- sqljdbc4.jar
And find example of how to connect to postgresql
````c#
DataFrame jdbcDf = spark.Read()
.Format("jdbc")
.Options(
new Dictionary
{
{"url", "jdbc:postgresql:192.168.10.198:5432/dbname"},
{"dbtable", "Roles"},
{"user", "postgres"},
{"password", "XXXX"},
{"ssl", "false"}
})
.Load();
````
Now I fix "no suitable driver" error with putting jar files but now I take this error :
> [Error] [JvmBridge] org.postgresql.ut
il.PSQLException: Connection to localhost:5432 refused. Check that the hostname
and port are correct and that the postmaster is accepting TCP/IP connections.
But I am not trying to connect localhost .
Contributor guide
Research direction
Start by reproducing the shown SparkSession JDBC calls with the listed .NET for Apache Spark, Spark, PostgreSQL, and JDBC versions. Inspect the JDBC driver and connection error, especially the URL and the reported localhost target. Done means documenting whether PostgreSQL connectivity works with the stated configuration or identifying the configuration issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, postgresql, spark
- Domain
- data, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100