aws / aws/sagemaker-spark

`import sagemaker_spark` failed on sagemaker notebook instance (platform identifier `notebook-al2-v1`)

Open
#144 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
301
Forks
129
PR merge metrics
No merged PRs in 30d

Description

Please fill out the form below.

### System Information
- **Spark or PySpark**: PySpark
- **SDK Version**: 1.4.2
- **Spark Version**: 2.4.0
- **Algorithm (e.g. KMeans)**: N/A

### Describe the problem
By following the code snippet from https://github.com/aws/sagemaker-spark/tree/master/sagemaker-pyspark-sdk#local-spark-on-sagemaker-notebook-instances to run local spark on sagemaker notebook instance (platform identifier `notebook-al2-v1`) `conda_python3` kernal, `import sagemaker_spark` failed. I started another sagemaker notebook instance (platform identifier `notebook-al1-v1`) it works well.

### Minimal repo / logs
Please provide any logs and a bare minimum reproducible test case, as this will be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

- **Exact command to reproduce**: `import sagemaker_pyspark`

traceback
```
import sagemaker_pyspark

from pyspark.sql import SparkSession

classpath = ":".join(sagemaker_pyspark.classpath_jars())

spark = SparkSession.builder.config("spark.driver.extraClassPath", classpath).getOrCreate()

---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/tmp/ipykernel_18565/3057191668.py in ()
----> 1 import sagemaker_pyspark
2 from pyspark.sql import SparkSession
3
4 classpath = ":".join(sagemaker_pyspark.classpath_jars())
5 spark = SparkSession.builder.config("spark.driver.extraClassPath", classpath).getOrCreate()

~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker_pyspark/__init__.py in
17 """
18
---> 19 from .wrapper import SageMakerJavaWrapper, Option
20 from .IAMRoleResource import IAMRole, IAMRoleFromConfig
21 from .SageMakerClients import SageMakerClients

~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker_pyspark/wrapper.py in
16 from abc import ABCMeta
17
---> 18 from pyspark import SparkContext
19 from pyspark.ml.common import _java2py
20 from pyspark.ml.wrapper import JavaWrapper

~/anaconda3/envs/python3/lib/python3.8/site-packages/pyspark/__init__.py in
49
50 from pyspark.conf import SparkConf
---> 51 from pyspark.context import SparkContext
52 from pyspark.rdd import RDD, RDDBarrier
53 from pyspark.files import SparkFiles

~/anaconda3/envs/python3/lib/python3.8/site-packages/pyspark/context.py in
29 from py4j.protocol import Py4JError
30
---> 31 from pyspark import accumulators
32 from pyspark.accumulators import Accumulator
33 from pyspark.broadcast import Broadcast, BroadcastPickleRegistry

~/anaconda3/envs/python3/lib/python3.8/site-packages/pyspark/accumulators.py in
95 import socketserver as SocketServer
96 import threading
---> 97 from pyspark.serializers import read_int, PickleSerializer
98
99

~/anaconda3/envs/python3/lib/python3.8/site-packages/pyspark/serializers.py in
69 xrange = range
70
---> 71 from pyspark import cloudpickle
72 from pyspark.util import _exception_message
73

~/anaconda3/envs/python3/lib/python3.8/site-packages/pyspark/cloudpickle.py in
143
144
--> 145 _cell_set_template_code = _make_cell_set_template_code()
146
147

~/anaconda3/envs/python3/lib/python3.8/site-packages/pyspark/cloudpickle.py in _make_cell_set_template_code()
124 )
125 else:
--> 126 return types.CodeType(
127 co.co_argcount,
128 co.co_kwonlyargcount,

TypeError: an integer is required (got type bytes)
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the import in the `conda_python3` environment on a `notebook-al2-v1` instance, then compare it with the working `notebook-al1-v1` setup. Inspect `sagemaker_pyspark/__init__.py`, `wrapper.py`, and the failing `pyspark/cloudpickle.py` path using SDK 1.4.2, Spark 2.4.0, and Python 3.8; done means `import sagemaker_pyspark` and the shown SparkSession setup complete successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.