AbsaOSS / AbsaOSS/ABRiS

Version 6.4.0 failing for Spark 3.5.0

Offen
#357 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Scala
Sterne
242
Forks
84
Ø Merge
10 Std. 12 Min.
Gemergte PRs (30 T.)
2

Beschreibung

Abris Version: 6.4.0
Spark Version: 3.5.0
Databricks Runtime: 14.3

Failing with error:
Cause: java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/SQLConf$LegacyBehaviorPolicy$
at org.apache.spark.sql.avro.AvroDeserializer.(AvroDeserializer.scala:62)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:39)
at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:37)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at scala.util.Failure.recover(Try.scala:234)
at org.apache.spark.sql.avro.AbrisAvroDeserializer.(AbrisAvroDeserializer.scala:37)

In Spark 3.4.0 "LegacyBehaviorPolicy" is available in same file as here:
https://github.com/apache/spark/blob/branch-3.4/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

However in Spark 3.5.0 "LegacyBehaviorPolicy" is available and used in package:
https://github.com/apache/spark/blob/branch-3.5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
But is available new package:
https://github.com/apache/spark/blob/branch-3.5/sql/api/src/main/scala/org/apache/spark/sql/internal/LegacyBehaviorPolicy.scala

Could this be issue for the above error?

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.