Dispose JvmBridge correctly
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
Currently, a `JvmBridge` instance is a static member of `SparkEnvironment` class. Without forcing the user to call something like `SparkEnvironment.JvmBridge.Dispose()` in his/her application, there is no clean way to dispose `JvmBridge`, thus Scala side handles the disconnect gracefully (#121).
One approach to address this issue is to have a ref-counted `SparkSession` where `JvmBridge.Dispose()` is called when the last `SparkSession` object is disposed.
- Note that the following should be handled:
```
using (var spark = SparkSession.Builder().GetOrCreate())
{
// do something
}
// New JvmBridge should be instantiated with the following.
using (var spark = SparkSession.Builder().GetOrCreate())
{
// do somthing
}
```
One issue with relying on `SparkSession` is that there are few classes such as `SparkConf` and `Builder` that accesses the `JvmBridge` directly from `SparkEnvironment` and these classes do not implement IDisposable (to be consistent with Scala Spark API), so it is harder to enforce cleaning up the `JvmBridge` if an user does the following
```
public static void Main(string[] args) {
var conf = new SparkConf();
// exits Main without creating SparkSession.
}
```
cc: @rapoth @stephentoub
Contributor guide
Assessment
This issue has not been assessed yet.