dotnet / dotnet/spark

[BUG]: When collected, long values are cast to int

Open
#1,155 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Describe the bug**
When running `DataFrame.Collect()` a long value in the dataset (even with a proper schema set to `LongType`) can be collected as an `int`.

**To Reproduce**

This reproduces the issue:
```c#
long value = 1;
string fieldName = "SomeLongValue";
var df = _spark.CreateDataFrame(
new[]
{
new GenericRow(new object[] { value } ),
},
new StructType(new [] { new StructField(fieldName, new LongType(), false)})
);
var collected = df.Collect().First().Get(fieldName);
Assert.Equal(value, collected); // this fails since `value` is a long whereas `collected` is an int
```

On the other hand, this test would run fine:
```c#
long value = long.MaxValue;
string fieldName = "SomeLongValue";
var df = _spark.CreateDataFrame(
new[]
{
new GenericRow(new object[] { value } ),
},
new StructType(new [] { new StructField(fieldName, new LongType(), false)})
);
var collected = df.Collect().First().Get(fieldName);
Assert.Equal(value, collected); // this works
```

It might come from Unpickling or from Spark -> Spark-Python, I have no idea.

**Expected behavior**

When the Schema specifies a column is of `LongType`, the collected value should be a `long`

**Desktop (please complete the following information):**
- OS: Windows 11
- Version: 2.1.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.