dotnet / dotnet/spark

[BUG]:When returning a StringDataFrameColumn from the apply method, a java.lang.IllegalArgumentException is thrown

Open
#1,231 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Describe the bug**
When returning a **StringDataFrameColumn** from the apply method, a **java.lang.IllegalArgumentException** is thrown.

**To Reproduce**

Steps to reproduce the behavior:
1.Implement an apply method that returns a **StringDataFrameColumn**.
2. Execute the code that triggers the apply method.
3. Observe that a **java.lang.IllegalArgumentException** is thrown at runtime

Minimal reproducible example:
- Spark version: 3.3.4
- Microsoft.Spark version: 2.3.0
- Microsoft.Spark.Worker: Microsoft.Spark.Worker.net8.0.win-x64-2.3.0

using Microsoft.Data.Analysis;
using Microsoft.Spark.Sql;
using Microsoft.Spark.Sql.Types;
using FxDataFrame = Microsoft.Data.Analysis.DataFrame;

namespace MySparkApp
{
class GroupApply
{
static void Main(string[] args)
{
SparkSession spark =
SparkSession
.Builder()
.AppName("group_apply_sample")
.GetOrCreate();

var df = spark.CreateDataFrame(new[]
{
new GenericRow(new object[] { 10, 2, 5.0, "this is description one" }),
new GenericRow(new object[] { 150, 10, 1.0, "this is description two" }),
new GenericRow(new object[] { 150, 3, 1.0 , "this is description three"}),
},
new StructType(new[]
{
new StructField("ProductId", new IntegerType()),
new StructField("Qty", new IntegerType()),
new StructField("Price", new DoubleType()),
new StructField("Description", new StringType())
}));

var returnSchema = new StructType(new[]
{
new StructField("Description", new StringType())
});

var result = df
.GroupBy("ProductId")
.Apply(returnSchema, dataFrame =>
{
var descriptions = new List();

foreach (var row in dataFrame.Rows)
{
int productId = (int)row["ProductId"];
string description = (string)row["Description"];

if (productId > 10)
{
descriptions.Add(description + productId);
}
}

var descriptionColumn = new StringDataFrameColumn("Description", descriptions);

var resultFrame = new FxDataFrame(descriptionColumn);

return resultFrame;
});

result.Show();

spark.Stop();
}
}
}

**Expected behavior**
The apply method should successfully return a **StringDataFrameColumn** without throwing an exception.

**Screenshots**
Image

**Desktop (please complete the following information):**
- OS: [Windows 10]
- Spark Version [3.3.4]
- Microsoft.Spark Version [2.3.0]
- Microsoft.Spark.Worker [Microsoft.Spark.Worker.net8.0.win-x64-2.3.0]

**Additional context**
This issue only occurs when the return type of the apply method is StringDataFrameColumn. Returning Int32DataFrameColumn and DoubleDataFrameColumn types will work as expected

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.