dotnet / dotnet/spark

[BUG]: Not able to run VectorUDF from Synapse notebook

Open
#1,120 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Describe the bug**
I am getting file not found '' when I follow the [VectorUDF example](https://github.com/dotnet/spark/blob/main/examples/Microsoft.Spark.CSharp.Examples/Sql/Batch/VectorDataFrameUdfs.cs) in Synapse Notebook.

**To Reproduce**

```csharp
//Cell 2
var names = new List { "john", "20" };

var df = spark.CreateDataFrame(
new List{new GenericRow(names.ToArray())},
new StructType(
new List()
{
new StructField("name", new StringType()),
new StructField("age", new StringType())
}));

//Cell 3
private static FxDataFrame CountCharacters(FxDataFrame dataFrame)
{
int characterCount = 0;

var characterCountColumn = new Int32DataFrameColumn("nameCharCount");
var ageColumn = new Int32DataFrameColumn("age");
ArrowStringDataFrameColumn nameColumn = dataFrame.Columns.GetArrowStringColumn("name");
for (long i = 0; i < dataFrame.Rows.Count; ++i)
{
characterCount += nameColumn[i].Length;
}

if (dataFrame.Rows.Count > 0)
{
characterCountColumn.Append(characterCount);
ageColumn.Append(dataFrame.Columns.GetInt32Column("age")[0]);
}

return new FxDataFrame(ageColumn, characterCountColumn);
}

//Cell 4
df.GroupBy("age")
.Apply(
new StructType(new[]
{
new StructField("age", new IntegerType()),
new StructField("nameCharCount", new IntegerType())
}),
r => CountCharacters(r))
.Show();

```
**Expected behavior**
Get a new dataframe with counts

**Screenshots**
![image](https://user-images.githubusercontent.com/16543422/200895085-fe763da6-1783-46ac-91d6-ece149eb3cc2.png)
![image](https://user-images.githubusercontent.com/16543422/200895161-8a6e9d3b-8b40-40bb-8715-4a33ca083567.png)

**Desktop (please complete the following information):**
- OS: Synapse Notebook
- Browser Edge
- Version

**Additional context**
I have not tried this in any other environment

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.