twosigma / twosigma/beakerx

Declare sparkSession and sparkContext as transient variables

Open
#8,165 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.9k
Forks
383
PR merge metrics
No merged PRs in 30d

Description

Spark magic `%%spark --start` does not declare spark context as a transient variable and due to this doing any sort of RDD manipulations will throw the below error

```
Caused by: java.io.NotSerializableException: org.apache.spark.SparkContext
Serialization stack:
- object not serializable (class: org.apache.spark.SparkContext, value: org.apache.spark.SparkContext@71836471)
- field (class: $iw, name: sc, type: class org.apache.spark.SparkContext)
```

the fix is to declare https://github.com/twosigma/beakerx/blob/d6233eb3baa8a8528e415401985da9e2e2d4ccfe/kernel/sparkex/src/main/java/com/twosigma/beakerx/widget/SparkEngineBase.java#L160 as transient variables

```
"@transient val %s = SparkVariable.getSparkSession()\n" +
"@transient val %s = %s.sparkContext\n" +
```

Please let me know if there is any other way to workaround this in the latest version

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.