Usage of Glue Data Catalog with sagemaker_pyspark
- Dominant language
- Scala
- Stars
- 301
- Forks
- 129
- PR merge metrics
- No merged PRs in 30d
Description
### System Information
- **Spark or PySpark**: PySpark
- **SDK Version**: v1.2.8
- **Spark Version**: v2.3.2
- **Algorithm (e.g. KMeans)**: n/a
### Describe the problem
I'm following the instructions proposed [HERE](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-glue.html) to connect a local spark session running in a notebook in Sagemaker to the Glue Data Catalog of my account.
I know this is doable via EMR but I'd like do to the same using a Sagemaker notebook (or any other kind of separate spark installation)
### Minimal repo / logs
Below is the current code that runs in the notebook but it doesn't actually work.
```
import sagemaker_pyspark
from pyspark.sql import SparkSession
classpath = ":".join(sagemaker_pyspark.classpath_jars())
spark = SparkSession.builder \
.config("spark.driver.extraClassPath", classpath) \
.config("hive.metastore.client.factory.class", "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory") \
.config("hive.metastore.schema.verification", "false") \
.enableHiveSupport() \
.getOrCreate()
```
Contributor guide
Research direction
Start by reproducing the provided SparkSession setup in a SageMaker notebook and compare it with the linked EMR Spark Glue instructions. The issue provides no logs, files, or tests; done would require a documented, reproducible way to connect the local session to the account's Glue Data Catalog, or a precise explanation of the unsupported setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100