Cannot run program "Microsoft.Spark.Worker": error=2, No such file or directory when using debug
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
I have setup dotnet Spark in MacOS by following [MacOS Getting Started](https://github.com/dotnet/spark/blob/main/docs/getting-started/macos-instructions.md). I am able to debug `MyJob` from Visual Studio by running below command,
spark-submit \
--class org.apache.spark.deploy.dotnet.DotnetRunner \
--master local \
~/csharp/MyJob/bin/Debug/net6.0/microsoft-spark-3-0_2.12-2.1.1.jar \
debug
When I added a UDF, I got below exception in the terminal window,
`Cannot run program "Microsoft.Spark.Worker": error=2, No such file or directory`
I was surprised and after trying different combinations, I figured out removing `debug` from spark-submit works as expected.
spark-submit \
--class org.apache.spark.deploy.dotnet.DotnetRunner \
--master local \
~/csharp/MyJob/bin/Debug/net6.0/microsoft-spark-3-0_2.12-2.1.1.jar \
dotnet ~/csharp/MyJob/bin/Debug/net6.0/MyJob.dll
Below is my environment variables,
export SPARK_HOME=~/spark-3.0.1-bin-hadoop2.7/
export DOTNET_WORKER_DIR=~/Microsoft.Spark.Worker-2.1.1/
export DOTNET_ASSEMBLY_SEARCH_PATHS=~/csharp/MyJob/bin/Debug/net6.0/
export DOTNET_WORKER_DEBUG=1
export PATH="$SPARK_HOME/bin:~/Microsoft.Spark.Worker-2.1.1/:/usr/local/opt/node@16/bin:$PATH"
Contributor guide
Assessment
This issue has not been assessed yet.