aws-samples / aws-samples/aws-glue-samples
Mark AWSGlueETL 4.0.0 as "provided" using SBT on EMR Release label 6.9.1, however GlueContext class not found during runtime
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 834
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 1
Description
Hi team, i want to use AWSGlueETL 4.0.0 on EMR 6.9.1 via Spark Scala.
As your code said that we should mark the libs as "provided".
The local testing looks good, the library can be downloaded and success run.
However when trying to run on the EMR, the logs said like this:
"java.lang.NoClassDefFoundError: com/amazonaws/services/glue/GlueContext"
Does EMR 6.9.1 indeed not provide the AWSGlueETL? if it is provided, could you help to fix that?
Thanks.
SBT: 1.5.3
Scala: 2.12.15
Spark: 3.3.0
AWSGlueETL: 4.0.0
```
ThisBuild / resolvers ++= {
Seq("AWS Glue Maven" at "https://aws-glue-etl-artifacts.s3.amazonaws.com/release/")}
val javaGlue = "com.amazonaws" % "AWSGlueETL" % "4.0.0" % "provided"
```
Contributor guide
Research direction
Start with the SBT dependency declaration shown in the issue and the EMR 6.9.1 runtime logs, then verify whether AWSGlueETL 4.0.0 is available and GlueContext can be resolved at runtime. Done means establishing whether the dependency is provided on this EMR release and documenting or correcting the required dependency setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, scala, spark
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100