ClassCastException error in geomesa_pyspark 4.0.5

ClassCastException error in geomesa_pyspark 4.0.5

I used pyspark to query data in hbase, and the code returned the following exception:

code is as follows:
import geomesa_pyspark conf = geomesa_pyspark.configure( jars=['/home/apache/geomesa-hbase-spark-runtime-hbase2_2.12-4.0.5.jar'], packages=['geomesa_pyspark', 'pytz'], spark_home='/opt/spark' ).setAppName('MyTest') from pyspark.sql import SparkSession spark = ( SparkSession .builder .config(conf=conf) .getOrCreate() ) params = { "hbase.catalog": "my_catalog", "hbase.zookeepers": "192.168.89.103:2181" } feature = "my_feature" df = ( spark .read .format("geomesa") .options(**params) .option("geomesa.feature", feature) .load() ) df.show()
the error message is as below:

java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
25/03/24 11:31:45 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job Traceback (most recent call last): File "pysparktest.py", line 33, in df.show() File "/opt/spark/python/pyspark/sql/dataframe.py", line 606, in show print(self._jdf.showString(n, 20, vertical)) File "/opt/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in call File "/opt/spark/python/pyspark/sql/utils.py", line 190, in deco return f(*a, **kw) File "/opt/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o73.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (sh03-stdspbigdata-dev-yarn02 executor 2): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)

Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-3461

Key: GEOMESA-3461
Type: Bug
Priority: Major
Status: To Do
Resolution: Unresolved
Assignee: Tomas Pulmano
Reporter: MikeZhou
Created: Sun, 23 Mar 2025 23:47:54 -0400
Updated: Thu, 27 Mar 2025 09:30:28 -0400

Comments:

557058:da7f6dfa-f0a1-468b-aec5-2cb61273feb5 (Thu, 27 Mar 2025 09:30:28 -0400):

Did you configure the custom serializers as outlined here? https://www.geomesa.org/documentation/stable/user/spark/core.html#simple-feature-serialization

Use the reported pyspark script, geomesa_pyspark.configure call, SparkSession setup, and GeoMesa HBase runtime JAR as the reproduction entry point. Check the custom serializer guidance linked in the comment and verify the Spark, Scala, and GeoMesa configuration involved. Done means identifying the incompatibility or required configuration and confirming that df.show() completes without the ClassCastException.