aws-samples / aws-samples/aws-glue-local-development
Runtime errors with Glue 4.0
- Dominant language
- Python
- Stars
- 34
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I am getting below runtime exceptions, when i tried to run glue job locally with Glue 4.0, AWS Corretto 20 (Java), Python 3.10.8, pytest 7.3.1.
Command:
../aws-glue-libs/bin/gluesparksubmit ./driver/main.py --JOB_NAME=movie_analytics_job --CONN_TYPE=file --INPUT_PATH=/Users//Documents/project/aws-glue-local-development/tests/samples --OUTPUT_PATH=/Users//Documents/project/aws-glue-local-development/tests/samples/results --MODULE_NAME=tasks --FUNCTION_NAME=create_and_stage_average_rating
spark-defaults.conf:
spark.driver.extraClassPath /Users//Documents/project/glue-libs/spark/jars/*:/Users//Documents/project/aws-glue-libs/jarsv1/*
spark.executor.extraClassPath /Users//Documents/project/glue-libs/spark/jars/*:/Users//Documents/project/aws-glue-libs/jarsv1/*
Exception:
java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.glue.util.StringToBoolean$
at com.amazonaws.services.glue.util.JsonOptions.extractWithDefaults(JsonOptions.scala:69)
at com.amazonaws.services.glue.util.JsonOptions.getOptions(JsonOptions.scala:48)
at com.amazonaws.services.glue.readers.CSVReader.liftedTree1$1(CSVReader.scala:57)
at com.amazonaws.services.glue.readers.CSVReader.(CSVReader.scala:57)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)
at com.amazonaws.services.glue.util.ClassUtils$.newInstanceByName(ClassUtils.scala:59)
at com.amazonaws.services.glue.readers.DynamicRecordStreamReader$.apply(DynamicRecordReader.scala:216)
at com.amazonaws.services.glue.hadoop.TapeHadoopRecordReaderSplittable.createDynamicRecordReader(TapeHadoopRecordReaderSplittable.scala:286)
at com.amazonaws.services.glue.hadoop.TapeHadoopRecordReaderSplittable.readHeader(TapeHadoopRecordReaderSplittable.scala:268)
at com.amazonaws.services.glue.hadoop.TapeHadoopRecordReaderSplittable.checkAndSetCSVSplittableHeader(TapeHadoopRecordReaderSplittable.scala:261)
at com.amazonaws.services.glue.hadoop.TapeHadoopRecordReaderSplittable.initialize(TapeHadoopRecordReaderSplittable.scala:200)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:221)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:218)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:173)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:72)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:106)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:138)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1623)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchMethodError: 'void org.json4s.CustomSerializer.(scala.Function1, scala.reflect.Manifest)' [in thread "Thread-4"]
pytests are passed though.
Appreciate if you can add instructions for running glue jobs locally as well.
Contributor guide
Assessment
This issue has not been assessed yet.