dotnet / dotnet/machinelearning

DataFrame doesn't decode boolean arrays correctly from Arrow

Open
#7,115 6 comments 1 reaction 0 assignees View on GitHub
area-DataFrame untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**System Information (please complete the following information):**
- OS & Version: Windows 10
- ML.NET Version: 0.21.1
- .NET Version: .NET 5.0

**Describe the bug**
Creating a dataframe from an arrow record batch where a column is a boolean array produces incorrect results (and occasionally even throws exceptions).

**To Reproduce**
Run:
```csharp
public void DataFrameDecodeBooleans()
{
BooleanArray boolArray = new BooleanArray.Builder().AppendNull().Append(false).Append(true).Build();
Field boolField = new Field("col", BooleanType.Default, true);
Schema schema = new Schema(new[] { boolField }, null);
RecordBatch batch = new RecordBatch(schema, new IArrowArray[] { boolArray }, boolArray.Length);
DataFrame df = DataFrame.FromArrowRecordBatch(batch);
DataFrameColumn boolCol = df["col"];
Assert.Equal(boolArray.Length, boolCol.Length);
for (int row = 0; row < boolArray.Length; row++)
{
Assert.Equal(boolArray.GetValue(row), boolCol[row]);
}
}
```

**Expected behavior**
Above test passes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.