aws / aws/sagemaker-spark

Usage of Glue Data Catalog with sagemaker_pyspark

Open
#109 16 comments 4 reactions 0 assignees View on GitHub
question
Dominant language
Scala
Stars
301
Forks
129
PR merge metrics
No merged PRs in 30d

Description

### System Information
- **Spark or PySpark**: PySpark
- **SDK Version**: v1.2.8
- **Spark Version**: v2.3.2
- **Algorithm (e.g. KMeans)**: n/a

### Describe the problem
I'm following the instructions proposed [HERE](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-glue.html) to connect a local spark session running in a notebook in Sagemaker to the Glue Data Catalog of my account.

I know this is doable via EMR but I'd like do to the same using a Sagemaker notebook (or any other kind of separate spark installation)

### Minimal repo / logs
Below is the current code that runs in the notebook but it doesn't actually work.
```
import sagemaker_pyspark
from pyspark.sql import SparkSession

classpath = ":".join(sagemaker_pyspark.classpath_jars())

spark = SparkSession.builder \
.config("spark.driver.extraClassPath", classpath) \
.config("hive.metastore.client.factory.class", "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory") \
.config("hive.metastore.schema.verification", "false") \
.enableHiveSupport() \
.getOrCreate()
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the provided SparkSession setup in a SageMaker notebook and compare it with the linked EMR Spark Glue instructions. The issue provides no logs, files, or tests; done would require a documented, reproducible way to connect the local session to the account's Glue Data Catalog, or a precise explanation of the unsupported setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.