dotnet / dotnet/spark

[BUG]: Better logging when UDF type can't be found

Open
#490 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Describe the bug**
When the SparkWorker.exe and the host runner .exe process load 2 different versions of an assembly containing a UDF, you get an error similar to:

```
ProcessStream() failed with exception: System.ArgumentNullException: Value cannot be null. (Parameter 'type')
at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)
at Microsoft.Spark.Utils.CommandSerDe.CreateUdfWrapperDelegate[T](Type type, Object[] parameters) in /_/src/csharp/Microsoft.Spark/Utils/CommandSerDe.cs:line 291
at Microsoft.Spark.Utils.CommandSerDe.Deserialize[T](Stream stream, SerializedMode& serializerMode, SerializedMode& deserializerMode, String& runMode) in /_/src/csharp/Microsoft.Spark/Utils/CommandSerDe.cs:line 243
at Microsoft.Spark.Worker.Processor.CommandProcessor.ReadSqlCommands(PythonEvalType evalType, Stream stream) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\CommandProcessor.cs:line 160
at Microsoft.Spark.Worker.Processor.CommandProcessor.ReadSqlCommands(PythonEvalType evalType, Stream stream, Version version) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\CommandProcessor.cs:line 117
at Microsoft.Spark.Worker.Processor.CommandProcessor.Process(Stream stream) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\CommandProcessor.cs:line 62
at Microsoft.Spark.Worker.Processor.PayloadProcessor.Process(Stream stream) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Processor\PayloadProcessor.cs:line 74
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 143
```

We should give a better error indicating what we were trying to do when we got an `ArgumentNullException` and what it means.

**To Reproduce**

Steps to reproduce the behavior:
1. Have the SparkWorker.exe and the host runner .exe load 2 different versions of an assembly containing a UDF. Usually the UDF type/method name has changed if you added more anonymous methods to a new version of an assembly.
2. Try to invoke the UDF
3. See error above.

**Expected behavior**
You should get an actionable error indicating what went wrong, and how you can fix it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.