dotnet / dotnet/spark

[FEATURE REQUEST]: Add Application .Zip file contents to Assembly Search Path

Open
#192 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Is your feature request related to a problem? Please describe.**
This is in relation to one of the new features added in v0.4.0:
> - The ability to use `--archives` option to deploy the worker binaries and assemblies that UDFs depend on (#187)

The `--archives` option only works for YARM setups. Databricks, for example, does not use YARM so the `--archives` parameter has no effect.

**Describe the solution you'd like**
For the application .zip file to be transferred to all workers, extracted and the contents to be in the assembly search path.

**Describe alternatives you've considered**
A partial workaround is adding the applications .zip file name to the `--files` parameter meets the first requirement of transferring it to all workers. However, it is not automatically extracted or added to the assembly search path.

Databricks Spark-Submit example:
```
["--files","DotNetSparkTest.zip","--class","org.apache.spark.deploy.dotnet.DotnetRunner","/dbfs/spark-dotnet/microsoft-spark-2.4.x-0.4.0.jar","/dbfs/apps/DotNetSparkTest.zip","DotNetSparkTest"]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.