dotnet / dotnet/spark

Cannot run program "Microsoft.Spark.Worker": error=2, No such file or directory when using debug

Open
#1,093 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

I have setup dotnet Spark in MacOS by following [MacOS Getting Started](https://github.com/dotnet/spark/blob/main/docs/getting-started/macos-instructions.md). I am able to debug `MyJob` from Visual Studio by running below command,

spark-submit \
--class org.apache.spark.deploy.dotnet.DotnetRunner \
--master local \
~/csharp/MyJob/bin/Debug/net6.0/microsoft-spark-3-0_2.12-2.1.1.jar \
debug

When I added a UDF, I got below exception in the terminal window,
`Cannot run program "Microsoft.Spark.Worker": error=2, No such file or directory`

I was surprised and after trying different combinations, I figured out removing `debug` from spark-submit works as expected.

spark-submit \
--class org.apache.spark.deploy.dotnet.DotnetRunner \
--master local \
~/csharp/MyJob/bin/Debug/net6.0/microsoft-spark-3-0_2.12-2.1.1.jar \
dotnet ~/csharp/MyJob/bin/Debug/net6.0/MyJob.dll

Below is my environment variables,

export SPARK_HOME=~/spark-3.0.1-bin-hadoop2.7/
export DOTNET_WORKER_DIR=~/Microsoft.Spark.Worker-2.1.1/
export DOTNET_ASSEMBLY_SEARCH_PATHS=~/csharp/MyJob/bin/Debug/net6.0/
export DOTNET_WORKER_DEBUG=1
export PATH="$SPARK_HOME/bin:~/Microsoft.Spark.Worker-2.1.1/:/usr/local/opt/node@16/bin:$PATH"

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.