aws-samples / aws-samples/aws-glue-local-development

Runtime errors with Glue 4.0

Open
#2 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34
Forks
10
PR merge metrics
No merged PRs in 30d

Description

Hi,

I am getting below runtime exceptions, when i tried to run glue job locally with Glue 4.0, AWS Corretto 20 (Java), Python 3.10.8, pytest 7.3.1.

Command:
../aws-glue-libs/bin/gluesparksubmit ./driver/main.py --JOB_NAME=movie_analytics_job --CONN_TYPE=file --INPUT_PATH=/Users//Documents/project/aws-glue-local-development/tests/samples --OUTPUT_PATH=/Users//Documents/project/aws-glue-local-development/tests/samples/results --MODULE_NAME=tasks --FUNCTION_NAME=create_and_stage_average_rating

spark-defaults.conf:
spark.driver.extraClassPath /Users//Documents/project/glue-libs/spark/jars/*:/Users//Documents/project/aws-glue-libs/jarsv1/*
spark.executor.extraClassPath /Users//Documents/project/glue-libs/spark/jars/*:/Users//Documents/project/aws-glue-libs/jarsv1/*

Exception:
java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.glue.util.StringToBoolean$
at com.amazonaws.services.glue.util.JsonOptions.extractWithDefaults(JsonOptions.scala:69)
at com.amazonaws.services.glue.util.JsonOptions.getOptions(JsonOptions.scala:48)
at com.amazonaws.services.glue.readers.CSVReader.liftedTree1$1(CSVReader.scala:57)
at com.amazonaws.services.glue.readers.CSVReader.(CSVReader.scala:57)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)
at com.amazonaws.services.glue.util.ClassUtils$.newInstanceByName(ClassUtils.scala:59)
at com.amazonaws.services.glue.readers.DynamicRecordStreamReader$.apply(DynamicRecordReader.scala:216)
at com.amazonaws.services.glue.hadoop.TapeHadoopRecordReaderSplittable.createDynamicRecordReader(TapeHadoopRecordReaderSplittable.scala:286)
at com.amazonaws.services.glue.hadoop.TapeHadoopRecordReaderSplittable.readHeader(TapeHadoopRecordReaderSplittable.scala:268)
at com.amazonaws.services.glue.hadoop.TapeHadoopRecordReaderSplittable.checkAndSetCSVSplittableHeader(TapeHadoopRecordReaderSplittable.scala:261)
at com.amazonaws.services.glue.hadoop.TapeHadoopRecordReaderSplittable.initialize(TapeHadoopRecordReaderSplittable.scala:200)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:221)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:218)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:173)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:72)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:106)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:138)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1623)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchMethodError: 'void org.json4s.CustomSerializer.(scala.Function1, scala.reflect.Manifest)' [in thread "Thread-4"]

pytests are passed though.

Appreciate if you can add instructions for running glue jobs locally as well.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.