aws / aws/amazon-sagemaker-examples

[Bug Report] Studio cannot use PySpark - Java gateway process exited before sending the driver its port number

Open
#2,872 0 comments 6 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
NA

**Describe the bug**
PySpark does not work in SageMaker Studio notebook.

**To reproduce**
1. Install java, set the JAVA_HOME environment variable.
```
# https://github.com/geerlingguy/ansible-role-java/issues/64
!mkdir -p /usr/share/man/man1

# https://stackoverflow.com/a/61902164/4281353
!apt update -y
!apt install software-properties-common -y
!apt install default-jdk -y

!java --version
!ls '/usr/lib/jvm/java-11-openjdk-amd64'
%env JAVA_HOME='/usr/lib/jvm/java-11-openjdk-amd64'

import os
os.environ['JAVA_HOME'] = '/usr/lib/jvm/java-11-openjdk-amd64'
```
Output:
```
openjdk 11.0.12 2021-07-20
OpenJDK Runtime Environment (build 11.0.12+7-post-Debian-2deb10u1)
OpenJDK 64-Bit Server VM (build 11.0.12+7-post-Debian-2deb10u1, mixed mode, sharing)
bin conf docs include jmods legal lib man release
env: JAVA_HOME='/usr/lib/jvm/java-11-openjdk-amd64'
```

2. Install pyspark, sagemaker_spark
```
!pip install pyspark sagemaker_pyspark
!pip show pyspark
```
Output:
```
Name: pyspark
Version: 3.1.2
Summary: Apache Spark Python API
Home-page: https://github.com/apache/spark/tree/master/python
Author: Spark Developers
Author-email: dev@spark.apache.org
License: http://www.apache.org/licenses/LICENSE-2.0
Location: /opt/conda/lib/python3.7/site-packages
Requires: py4j
Required-by: sagemaker-pyspark
```

3. Invoke PySpark
Run below and it causes the error in the log.
```
%env SPARK_VERSION='3.1.2'

from pyspark.sql import SparkSession, Row, DataFrame
import json
import pandas as pd
import sagemaker_pyspark

import pydeequ

classpath = ":".join(sagemaker_pyspark.classpath_jars())
spark = (SparkSession
.builder
.config("spark.driver.extraClassPath", classpath)
.config("spark.jars.packages", pydeequ.deequ_maven_coord)
.config("spark.jars.excludes", pydeequ.f2j_maven_coord)
.getOrCreate())
```

**Logs**

```
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
in
11 .config("spark.driver.extraClassPath", classpath)
12 .config("spark.jars.packages", pydeequ.deequ_maven_coord)
---> 13 .config("spark.jars.excludes", pydeequ.f2j_maven_coord)
14 .getOrCreate())

/opt/conda/lib/python3.7/site-packages/pyspark/sql/session.py in getOrCreate(self)
226 sparkConf.set(key, value)
227 # This SparkContext may be an existing one.
--> 228 sc = SparkContext.getOrCreate(sparkConf)
229 # Do not update `SparkConf` for existing `SparkContext`, as it's shared
230 # by all sessions.

/opt/conda/lib/python3.7/site-packages/pyspark/context.py in getOrCreate(cls, conf)
382 with SparkContext._lock:
383 if SparkContext._active_spark_context is None:
--> 384 SparkContext(conf=conf or SparkConf())
385 return SparkContext._active_spark_context
386

/opt/conda/lib/python3.7/site-packages/pyspark/context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
142 " is not allowed as it is a security risk.")
143
--> 144 SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
145 try:
146 self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

/opt/conda/lib/python3.7/site-packages/pyspark/context.py in _ensure_initialized(cls, instance, gateway, conf)
329 with SparkContext._lock:
330 if not SparkContext._gateway:
--> 331 SparkContext._gateway = gateway or launch_gateway(conf)
332 SparkContext._jvm = SparkContext._gateway.jvm
333

/opt/conda/lib/python3.7/site-packages/pyspark/java_gateway.py in launch_gateway(conf, popen_kwargs)
106
107 if not os.path.isfile(conn_info_file):
--> 108 raise Exception("Java gateway process exited before sending its port number")
109
110 with open(conn_info_file, "rb") as info:

Exception: Java gateway process exited before sending its port number
```

# Related

* [Exception: Java gateway process exited before sending the driver its port number #743](https://github.com/jupyter/notebook/issues/743)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.