dotnet / dotnet/spark

[BUG]: spark.Dispose() and spark.Stop() throw error

Open
#964 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Describe the bug**
spark.Dispose() and spark.Stop() does not work and throw error.

**To Reproduce**

Steps to reproduce the behavior:
1. Go to Azure Synapse Studio
2. Start Spark .NET Session.
3. Try to execute spark.Dispose() or spark.Stop()
4. Session does not stop and throws below error.
If you think the bug depends on external factors (e.g., dataset), please provide us with a minimal reproducible example that consists of the following items:

- a minimal dataset, necessary to reproduce the error
- the minimal runnable code necessary to reproduce the error, which can be run on the given dataset
- the necessary information on any used packages, .NET runtime version, and system it is run on
- in the case of random processes, a seed for reproducibility

**Expected behavior**
Spark context should be stopped correctly.

**Screenshots**
![image](https://user-images.githubusercontent.com/31816865/131150271-e26221c1-930d-44a3-93ba-e5a666b8474e.png)

**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]

**Additional context**
Error message: ```
[2021-08-27T15:15:18.6083607Z] [07759f195a294a98b5ac3fc4a7e8522800554b46593] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'addFile' failed for class '50' when called with 2 arguments ([Index=1, Type=String, Value=/mnt/var/hadoop/tmp/nm-local-dir/usercache/trusted-service-user/appcache/application_1630075737145_0002/container_1630075737145_0002_01_000001/.sparkdotnet/.dotnetinteractive/mqi1almc.zjx/4744a673-f62f-4fd9-bc6e-5c84ec6d4f97-1-20.dll], [Index=2, Type=Boolean, Value=False], )
[2021-08-27T15:15:18.6092818Z] [07759f195a294a98b5ac3fc4a7e8522800554b46593] [Error] [JvmBridge] java.lang.NullPointerException
at org.apache.spark.SparkFiles$.getRootDirectory(SparkFiles.scala:37)
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1568)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:159)
at org.apache.spark.api.dotnet.DotnetBackendHandler$$anonfun$handleBackendRequest$1.apply$mcV$sp(DotnetBackendHandler.scala:99)
at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[2021-08-27T15:15:18.6096229Z] [07759f195a294a98b5ac3fc4a7e8522800554b46593] [Exception] [JvmBridge] JVM method execution failed: Nonstatic method 'addFile' failed for class '50' when called with 2 arguments ([Index=1, Type=String, Value=/mnt/var/hadoop/tmp/nm-local-dir/usercache/trusted-service-user/appcache/application_1630075737145_0002/container_1630075737145_0002_01_000001/.sparkdotnet/.dotnetinteractive/mqi1almc.zjx/4744a673-f62f-4fd9-bc6e-5c84ec6d4f97-1-20.dll], [Index=2, Type=Boolean, Value=False], )
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
System.Exception: JVM method execution failed: Nonstatic method 'addFile' failed for class '50' when called with 2 arguments ([Index=1, Type=String, Value=/mnt/var/hadoop/tmp/nm-local-dir/usercache/trusted-service-user/appcache/application_1630075737145_0002/container_1630075737145_0002_01_000001/.sparkdotnet/.dotnetinteractive/mqi1almc.zjx/4744a673-f62f-4fd9-bc6e-5c84ec6d4f97-1-20.dll], [Index=2, Type=Boolean, Value=False], )
---> Microsoft.Spark.JvmException: java.lang.NullPointerException
at org.apache.spark.SparkFiles$.getRootDirectory(SparkFiles.scala:37)
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1568)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:159)
at org.apache.spark.api.dotnet.DotnetBackendHandler$$anonfun$handleBackendRequest$1.apply$mcV$sp(DotnetBackendHandler.scala:99)
at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

--- End of inner exception stack trace ---
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object arg0, Object arg1)
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName, Object arg0, Object arg1)
at Microsoft.Spark.Interop.Ipc.JvmObjectReference.Invoke(String methodName, Object arg0, Object arg1)
at Microsoft.Spark.SparkContext.AddFile(String path, Boolean recursive)
at Microsoft.Spark.Extensions.DotNet.Interactive.AssemblyKernelExtension.<>c__DisplayClass2_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.DotNet.Interactive.KernelCommandPipeline.<>c__DisplayClass6_1.<b__3>d.MoveNext() in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 75
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.DotNet.Interactive.Kernel.b__21_0(KernelCommand originalCommand, KernelInvocationContext context, KernelPipelineContinuation next) in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\Kernel.cs:line 126
at Microsoft.DotNet.Interactive.KernelCommandPipeline.<>c__DisplayClass6_1.<b__3>d.MoveNext() in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 75
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.DotNet.Interactive.Kernel.SetKernel(KernelCommand command, KernelInvocationContext context, KernelPipelineContinuation next) in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\Kernel.cs:line 218
at Microsoft.DotNet.Interactive.KernelCommandPipeline.<>c__DisplayClass6_1.<b__3>d.MoveNext() in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 75
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.DotNet.Interactive.Kernel.<>c.<b__20_0>d.MoveNext() in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\Kernel.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.DotNet.Interactive.KernelCommandPipeline.<>c__DisplayClass6_0.<g__Combine|2>d.MoveNext() in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 76
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.DotNet.Interactive.KernelCommandPipeline.<>c__DisplayClass6_0.<g__Combine|2>d.MoveNext() in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 76
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.DotNet.Interactive.KernelCommandPipeline.<>c__DisplayClass6_0.<g__Combine|2>d.MoveNext() in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 76
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.DotNet.Interactive.KernelCommandPipeline.<>c__DisplayClass6_0.<g__Combine|2>d.MoveNext() in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 76
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.DotNet.Interactive.KernelCommandPipeline.SendAsync(KernelCommand command, KernelInvocationContext context) in F:\workspace\_work\1\s\src\Microsoft.DotNet.Interactive\KernelCommandPipeline.cs:line 42
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.