[BUG]: Spark has to be restarted if using UDFs and recompiling
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
Whenever I use UDFs in an application utilizing .NET for Spark, and recompile the application regardless of whether I change the UDF code or not, the Spark worker no longer manages to load the UDF when it has to be executed as part of a query. The only workaround I found is to restart the Spark process before each run. This is OK when doing data science on a desktop, but not so if using Spark for server side processing where one expects Spark to keep running when updating the binaries that leverage Spark (there my be old and new instances during a roll-out for example). This is consistently reproducible for me when running Spark in debug mode (as a standalone server) regardless of which UDF I use or which dataset I use.
**Desktop (please complete the following information):**
- OS: Linux x64 (Ubuntu 19.10)
Contributor guide
Assessment
This issue has not been assessed yet.