aws-samples / aws-samples/aws-glue-samples

Mark AWSGlueETL 4.0.0 as "provided" using SBT on EMR Release label 6.9.1, however GlueContext class not found during runtime

Open
#171 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.5k
Forks
834
Avg merge
2d 17h
Merged PRs (30d)
1

Description

Hi team, i want to use AWSGlueETL 4.0.0 on EMR 6.9.1 via Spark Scala.

As your code said that we should mark the libs as "provided".
The local testing looks good, the library can be downloaded and success run.
However when trying to run on the EMR, the logs said like this:
"java.lang.NoClassDefFoundError: com/amazonaws/services/glue/GlueContext"

Does EMR 6.9.1 indeed not provide the AWSGlueETL? if it is provided, could you help to fix that?

Thanks.

SBT: 1.5.3
Scala: 2.12.15
Spark: 3.3.0
AWSGlueETL: 4.0.0

```
ThisBuild / resolvers ++= {
Seq("AWS Glue Maven" at "https://aws-glue-etl-artifacts.s3.amazonaws.com/release/")}

val javaGlue = "com.amazonaws" % "AWSGlueETL" % "4.0.0" % "provided"

```

Contributor guide

Open the contributing guide

Research direction

Start with the SBT dependency declaration shown in the issue and the EMR 6.9.1 runtime logs, then verify whether AWSGlueETL 4.0.0 is available and GlueContext can be resolved at runtime. Done means establishing whether the dependency is provided on this EMR release and documenting or correcting the required dependency setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, scala, spark
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.