Error while using JDBC2S jar with PySpark
- Ngôn ngữ chính
- Scala
- Star
- 10
- Fork
- 3
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
Hi Jdbc2S team,
Thank you so much for this repo.
I'm working on a personal project where I need to fetch real-time data from the Database using Spark Structured Streaming API.
I tried 2 approaches listed below.
1. Downloaded the Jdbc2S jar and used it during SparkSession initiation.
```
format = "za.co.absa.spark.jdbc.streaming.source.providers.JDBCStreamingSourceProviderV1"
spark = SparkSession.builder.config("spark.jars","./pathtoJar1, ./jdbc2s_2.11-1.2.0.jar").appName("StreamingCount").getOrCreate()
```
When I try to execute the script, I get the following error.
```
Traceback (most recent call last):
File "SparkStreaming.py", line 14, in
count_sql = spark.readStream.format(format) \
File "/Users/subramaniav1/Library/Python/3.8/lib/python/site-packages/pyspark/sql/streaming.py", line 454, in load
return self._df(self._jreader.load())
File "/Users/subramaniav1/Library/Python/3.8/lib/python/site-packages/py4j/java_gateway.py", line 1321, in __call__
return_value = get_return_value(
File "/Users/subramaniav1/Library/Python/3.8/lib/python/site-packages/pyspark/sql/utils.py", line 111, in deco
return f(*a, **kw)
File "/Users/subramaniav1/Library/Python/3.8/lib/python/site-packages/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o37.load.
: java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/DataSourceV2
```
I use Python 3.8.9, Scala 2.12.15 and Spark 3.2.1
2. I tried to follow the steps mentioned in https://stackoverflow.com/a/72158761 where someone was able to modify and use this with PySpark. I modified the object JDBCStreamingSourceV1 in JDBCStreamingSourceV1.scala as mentioned, and the pom.xml with updated scala version and tried to rebuild the jar using Maven.
I got build failure with the following error
```
Run starting. Expected test count is: 29
TestClassJDBCSingleOffset:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/reader/streaming/Offset
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1013)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at za.co.absa.spark.jdbc.streaming.source.offsets.TestClassJDBCSingleOffset.$anonfun$new$1(TestClassJDBCSingleOffset.scala:28)
...
Cause: java.lang.ClassNotFoundException: org.apache.spark.sql.sources.v2.reader.streaming.Offset
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1013)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
```
It'd be of great help if you can help with steps to use this repo with Pyspark
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.