dotnet / dotnet/spark

Dispose JvmBridge correctly

Open
#24 1 comment 0 reactions 0 assignees View on GitHub
good first issue help wanted
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

Currently, a `JvmBridge` instance is a static member of `SparkEnvironment` class. Without forcing the user to call something like `SparkEnvironment.JvmBridge.Dispose()` in his/her application, there is no clean way to dispose `JvmBridge`, thus Scala side handles the disconnect gracefully (#121).

One approach to address this issue is to have a ref-counted `SparkSession` where `JvmBridge.Dispose()` is called when the last `SparkSession` object is disposed.
- Note that the following should be handled:
```
using (var spark = SparkSession.Builder().GetOrCreate())
{
// do something
}

// New JvmBridge should be instantiated with the following.
using (var spark = SparkSession.Builder().GetOrCreate())
{
// do somthing
}
```

One issue with relying on `SparkSession` is that there are few classes such as `SparkConf` and `Builder` that accesses the `JvmBridge` directly from `SparkEnvironment` and these classes do not implement IDisposable (to be consistent with Scala Spark API), so it is harder to enforce cleaning up the `JvmBridge` if an user does the following
```
public static void Main(string[] args) {
var conf = new SparkConf();
// exits Main without creating SparkSession.
}

```

cc: @rapoth @stephentoub

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.