Error from many DataFrame methods after UDF called in DataFrame.WithColumn
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
I'm a long time C# programmer but just getting my feet wet with .Net for Apache Spark. Following many "getting started" instructions and videos, I installed:
7-Zip
Java 8
I downloaded Apache Spark from https://spark.apache.org/downloads.html
.NET for Apache Spark v2.1.1
WinUtils.exe I'm running this on Window 10
Problem:
When I call DataFrame.Show() after doing a DataFrame.WithColumn() using a UDF, I always get an error: [2023-02-07T15:45:31.3903664Z] [DESKTOP-H37P8Q0] **[Error] [TaskRunner] [0] ProcessStream() failed with exception: System.ArgumentNullException: Value cannot be null. Parameter name: type**
TestCases.csv looks like this:
[TestCases.csv](https://github.com/dotnet/spark/files/10687418/TestCases.csv)
OrderList.csv looks like this:
[OrderList.csv](https://github.com/dotnet/spark/files/10687427/OrderList.csv)
Here is the Program class of the TestSparkApp console project:
[Program.cs.txt](https://github.com/dotnet/spark/files/10687578/Program.cs.txt)
and supporting classes:
[Player.cs.txt](https://github.com/dotnet/spark/files/10687503/Player.cs.txt)
[Collector.cs.txt](https://github.com/dotnet/spark/files/10687506/Collector.cs.txt)
Here is the output of the above app:
[TestSpartAppOutput.txt](https://github.com/dotnet/spark/files/10687571/TestSpartAppOutput.txt)
Note that the same bug will appear executing many different methods on the DataFrame object but only after a call to the WithColumn method using a UDF. In this case, the code looks like this:
```
// user defined function
Func GetSubst = Udf(
(strOrder, strPlayers) =>
{
return GetSubstance(strOrder, strPlayers);
});
// call the user defined function and add a new column to the dataframe
ordersFrame = ordersFrame.WithColumn("substance", GetSubst(ordersFrame["names"], ordersFrame["players"]).Cast("Integer"));
// *** This is where the error will be thrown, but if I comment it out, the same error will be thrown later
// print out the data
ordersFrame.Show(20, 20, false);
```
however, I've tried it with other UDFs followed by other DataFrame method calls and I always get the same error. In the Main() function, you will see a later foreach loop. If I comment out the ordersFrame.Show() call, and comment in the contents of the loop, I will get the same error when I access row.Values[0].ToString().
I wonder if I have missed something in my installation?
**Desktop (please complete the following information):**
- OS: Windows 10
- Browser n/a
- Version see above
Contributor guide
Assessment
This issue has not been assessed yet.