dotnet / dotnet/spark

[BUG]: .Net Interactive: UDF use deprecated BinaryFormatter, fail in VS Code

Open
#795 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Describe the bug**
UDFs are serialized using BinaryFormatter, which is deprecated for security reasons.

In environments where `BinaryFormatter` is already disabled, this causes UDFs to fail with a `NotSupportedException`.

**To Reproduce**

Steps to reproduce the behavior:
1. Use Visual Studio Code with the [.Net Interactive Notebooks](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.dotnet-interactive-vscode) extension
2. Follow the steps in [UDFs in .NET Interactive](https://docs.microsoft.com/en-us/dotnet/spark/how-to-guides/dotnet-interactive-udf-issue#define-a-udf-in-net-interactive)
3. Executing cell #3 (which contains the definition of the UDF) fails with:

```
Error: System.NotSupportedException: BinaryFormatter serialization and deserialization are disabled within this application. See https://aka.ms/binaryformatter for more information.
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph)
at Microsoft.Spark.Utils.CommandSerDe.Serialize(Delegate func, SerializedMode deserializerMode, SerializedMode serializerMode)
at Microsoft.Spark.Sql.Functions.CreateUdf(String name, Delegate execute, PythonEvalType evalType, String returnType)
at Microsoft.Spark.Sql.Functions.CreateUdf[TResult](String name, Delegate execute, PythonEvalType evalType)
at Microsoft.Spark.Sql.Functions.CreateUdf[TResult](String name, Delegate execute)
at Microsoft.Spark.Sql.Functions.Udf[T,TResult](Func`2 udf)
at Submission#11.<>d__0.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)
```

**Expected behavior**

- Should be able to use UDFs
- The example in [UDFs in .NET Interactive](https://docs.microsoft.com/en-us/dotnet/spark/how-to-guides/dotnet-interactive-udf-issue#define-a-udf-in-net-interactive) should work
- UDFs should not use `BinaryFormatter`

**Screenshots**
NA

**Desktop (please complete the following information):**
- OS: macOS 10.15.7
- .NET 5 (5.0.100)
- Apache Spark 3.0.1
- .NET for Apache Spark v1.0.0 (netcoreapp3.1, macOS)
- VS Code:
- Version: 1.53.0-insider
- Commit: 96b426ef1da0f0a51ce9ef1931cd1fd2322547e4
- Date: 2020-12-14T05:49:45.285Z
- Electron: 11.0.3
- Chrome: 87.0.4280.67
- Node.js: 12.18.3
- V8: 8.7.220.25-electron.0
- OS: Darwin x64 19.6.0
- .Net Interactive Notebooks v1.0.160901

**Additional context**
NA

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.