Scala Spark Magic not able to identify spark master for mesos
- Dominant language
- Jupyter Notebook
- Stars
- 2.9k
- Forks
- 383
- PR merge metrics
- No merged PRs in 30d
Description
I am using Jupyter Notebook to run Scala Spark Magic. Here are the config JSON and Notebook I am using.
[beakerx-notebook.txt](https://github.com/twosigma/beakerx/files/3399911/beakerx-notebook.txt)
[beakerx-json.txt](https://github.com/twosigma/beakerx/files/3399912/beakerx-json.txt)
The second cell in the attached notebook, which invoke `%%spark --start` is throwing error:
`2019-07-17 02:40:06:849 +0000 [Thread-0] ERROR SparkContext - Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: 'mesos://zk://zk-1.zk:2181,zk-2.zk:2181,zk-3.zk:2181,zk-4.zk:2181,zk-5.zk:2181/mesos'`
Here is the complete log:
[beakerx-error.log](https://github.com/twosigma/beakerx/files/3399916/beakerx-error.log)
I don't know what I am doing wrong because following notebook is working fine:
[beakerx-working-notebook.txt](https://github.com/twosigma/beakerx/files/3399934/beakerx-working-notebook.txt)
It is also working if we specify `spark.master=local[*]`.
Using Spark version 2.4.
It seems Spark is not able to load correct `ExternalClusterManager`.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.