twosigma / twosigma/beakerx

Scala Spark Magic not able to identify spark master for mesos

Open
#8,094 2 comments 0 reactions 1 assignee Claimed by @LeeTZ View on GitHub
Dominant language
Jupyter Notebook
Stars
2.9k
Forks
383
PR merge metrics
No merged PRs in 30d

Description

I am using Jupyter Notebook to run Scala Spark Magic. Here are the config JSON and Notebook I am using.

[beakerx-notebook.txt](https://github.com/twosigma/beakerx/files/3399911/beakerx-notebook.txt)
[beakerx-json.txt](https://github.com/twosigma/beakerx/files/3399912/beakerx-json.txt)

The second cell in the attached notebook, which invoke `%%spark --start` is throwing error:

`2019-07-17 02:40:06:849 +0000 [Thread-0] ERROR SparkContext - Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: 'mesos://zk://zk-1.zk:2181,zk-2.zk:2181,zk-3.zk:2181,zk-4.zk:2181,zk-5.zk:2181/mesos'`

Here is the complete log:
[beakerx-error.log](https://github.com/twosigma/beakerx/files/3399916/beakerx-error.log)

I don't know what I am doing wrong because following notebook is working fine:
[beakerx-working-notebook.txt](https://github.com/twosigma/beakerx/files/3399934/beakerx-working-notebook.txt)

It is also working if we specify `spark.master=local[*]`.

Using Spark version 2.4.

It seems Spark is not able to load correct `ExternalClusterManager`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.