aws / aws/amazon-sagemaker-examples
[Bug Report] Studio cannot use PySpark - Java gateway process exited before sending the driver its port number
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
NA
**Describe the bug**
PySpark does not work in SageMaker Studio notebook.
**To reproduce**
1. Install java, set the JAVA_HOME environment variable.
```
# https://github.com/geerlingguy/ansible-role-java/issues/64
!mkdir -p /usr/share/man/man1
# https://stackoverflow.com/a/61902164/4281353
!apt update -y
!apt install software-properties-common -y
!apt install default-jdk -y
!java --version
!ls '/usr/lib/jvm/java-11-openjdk-amd64'
%env JAVA_HOME='/usr/lib/jvm/java-11-openjdk-amd64'
import os
os.environ['JAVA_HOME'] = '/usr/lib/jvm/java-11-openjdk-amd64'
```
Output:
```
openjdk 11.0.12 2021-07-20
OpenJDK Runtime Environment (build 11.0.12+7-post-Debian-2deb10u1)
OpenJDK 64-Bit Server VM (build 11.0.12+7-post-Debian-2deb10u1, mixed mode, sharing)
bin conf docs include jmods legal lib man release
env: JAVA_HOME='/usr/lib/jvm/java-11-openjdk-amd64'
```
2. Install pyspark, sagemaker_spark
```
!pip install pyspark sagemaker_pyspark
!pip show pyspark
```
Output:
```
Name: pyspark
Version: 3.1.2
Summary: Apache Spark Python API
Home-page: https://github.com/apache/spark/tree/master/python
Author: Spark Developers
Author-email: dev@spark.apache.org
License: http://www.apache.org/licenses/LICENSE-2.0
Location: /opt/conda/lib/python3.7/site-packages
Requires: py4j
Required-by: sagemaker-pyspark
```
3. Invoke PySpark
Run below and it causes the error in the log.
```
%env SPARK_VERSION='3.1.2'
from pyspark.sql import SparkSession, Row, DataFrame
import json
import pandas as pd
import sagemaker_pyspark
import pydeequ
classpath = ":".join(sagemaker_pyspark.classpath_jars())
spark = (SparkSession
.builder
.config("spark.driver.extraClassPath", classpath)
.config("spark.jars.packages", pydeequ.deequ_maven_coord)
.config("spark.jars.excludes", pydeequ.f2j_maven_coord)
.getOrCreate())
```
**Logs**
```
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
in
11 .config("spark.driver.extraClassPath", classpath)
12 .config("spark.jars.packages", pydeequ.deequ_maven_coord)
---> 13 .config("spark.jars.excludes", pydeequ.f2j_maven_coord)
14 .getOrCreate())
/opt/conda/lib/python3.7/site-packages/pyspark/sql/session.py in getOrCreate(self)
226 sparkConf.set(key, value)
227 # This SparkContext may be an existing one.
--> 228 sc = SparkContext.getOrCreate(sparkConf)
229 # Do not update `SparkConf` for existing `SparkContext`, as it's shared
230 # by all sessions.
/opt/conda/lib/python3.7/site-packages/pyspark/context.py in getOrCreate(cls, conf)
382 with SparkContext._lock:
383 if SparkContext._active_spark_context is None:
--> 384 SparkContext(conf=conf or SparkConf())
385 return SparkContext._active_spark_context
386
/opt/conda/lib/python3.7/site-packages/pyspark/context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
142 " is not allowed as it is a security risk.")
143
--> 144 SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
145 try:
146 self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
/opt/conda/lib/python3.7/site-packages/pyspark/context.py in _ensure_initialized(cls, instance, gateway, conf)
329 with SparkContext._lock:
330 if not SparkContext._gateway:
--> 331 SparkContext._gateway = gateway or launch_gateway(conf)
332 SparkContext._jvm = SparkContext._gateway.jvm
333
/opt/conda/lib/python3.7/site-packages/pyspark/java_gateway.py in launch_gateway(conf, popen_kwargs)
106
107 if not os.path.isfile(conn_info_file):
--> 108 raise Exception("Java gateway process exited before sending its port number")
109
110 with open(conn_info_file, "rb") as info:
Exception: Java gateway process exited before sending its port number
```
# Related
* [Exception: Java gateway process exited before sending the driver its port number #743](https://github.com/jupyter/notebook/issues/743)
Contributor guide
Assessment
This issue has not been assessed yet.