dotnet / dotnet/spark

[BUG]: Spark has to be restarted if using UDFs and recompiling

Open
#426 13 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

Whenever I use UDFs in an application utilizing .NET for Spark, and recompile the application regardless of whether I change the UDF code or not, the Spark worker no longer manages to load the UDF when it has to be executed as part of a query. The only workaround I found is to restart the Spark process before each run. This is OK when doing data science on a desktop, but not so if using Spark for server side processing where one expects Spark to keep running when updating the binaries that leverage Spark (there my be old and new instances during a roll-out for example). This is consistently reproducible for me when running Spark in debug mode (as a standalone server) regardless of which UDF I use or which dataset I use.

**Desktop (please complete the following information):**
- OS: Linux x64 (Ubuntu 19.10)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.