dotnet / dotnet/machinelearning

ToDataFrame(): Array dimensions exceeded supported range.

Open
#5,869 1 comment 0 reactions 1 assignee Claimed by @pgovind View on GitHub
area-DataFrame
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**System Information (please complete the following information):**
- OS & Version: Windows Server 2019
- ML.NET Version: ML.NET v1.5.5
- .NET Version: .NET 5.0

**Describe the bug**
I am trying to run ML on a data set with 168 421 466 rows and 621 columns.
During the run, I get exception "Array dimensions exceeded supported range".

Snippet from error log with links to Github locations:
```
Array dimensions exceeded supported range.
at System.Collections.Generic.List`1.set_Capacity(Int32 value)
at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
at System.Collections.Generic.List`1.Add(T item)
at Microsoft.Data.Analysis.StringDataFrameColumn.Append(String value)
https://github.com/dotnet/machinelearning/blob/6114b4f960ed8d133e03d5798b21f38fb91e848a/src/Microsoft.Data.Analysis/StringDataFrameColumn.cs#L66

at Microsoft.Data.Analysis.StringDataFrameColumn.AddValueUsingCursor(DataViewRowCursor cursor, Delegate getter)
https://github.com/dotnet/machinelearning/blob/04dda55ab0902982b16309c8e151f13a53e9366d/src/Microsoft.Data.Analysis/IDataView.Extension.cs#L135

at Microsoft.ML.IDataViewExtensions.ToDataFrame(IDataView dataView, Int64 maxRows, String[] selectColumns)
```

**Question**
I am trying to understand why the supported range is exceeded, and what is the range limit?

My current understanding is as follows:
- dataFrameColumns: List has all of the columns
- Each DataFrameColumn includes all the values from each row, for that column
- a buffer List> is used here to bypass int.MaxValue according to comment on this line https://github.com/dotnet/machinelearning/blob/6114b4f960ed8d133e03d5798b21f38fb91e848a/src/Microsoft.Data.Analysis/StringDataFrameColumn.cs#L20
- The string is then added here to the inner List until it is full https://github.com/dotnet/machinelearning/blob/6114b4f960ed8d133e03d5798b21f38fb91e848a/src/Microsoft.Data.Analysis/StringDataFrameColumn.cs#L64-L72
_edit:_ It seems like a reasonable way to raise the maximum count to (int.MaxValue * int.MaxValue)

In my case, I have less than int.MaxValue rows. So, there should be only on inner List. Alternatively, it could a restriction of 2 GB object size. Per my understanding each string reference takes 8 bytes, and it would hit the limit at 268 million strings. It is close to my 168 million rows, but there is still quite some margin left. .NET Core also should not have the 2GB per object limit according to https://github.com/dotnet/coreclr/pull/8853

**Additional context*
For full log file see https://github.com/dotnet/machinelearning-modelbuilder/issues/1561

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.