dotnet / dotnet/machinelearning

Easier to create `DataFrame` from `DataViewSchema`

Open
#6,473 2 comments 0 reactions 0 assignees View on GitHub
area-DataFrame enhancement
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**Is your feature request related to a problem? Please describe.**
[Loading a model](https://learn.microsoft.com/en-us/dotnet/api/microsoft.ml.modeloperationscatalog.load?view=ml-dotnet) returns a `ITransformer` and its input column information in a `DataViewSchema`. `IDataView` is required for making predictions and creating a `DataFrame` should be the simplest way, especially when it provides [`Append`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.data.analysis.dataframe.append?view=ml-dotnet-preview) to add data row by row. Unfortunately, creating a `DataFrame` from a `DataViewSchema` is currently not easy because the `DataFrame` constructors basically only accepts `IEnumerable` and there is no direct conversion from `DataViewSchema` to `IEnumerable`.

**Describe the solution you'd like**
`IDataViewExtensions.ToDataFrame` actually contains the logic converting `DataViewSchema` to `IEnumerable`. To solve this issue, that part of the code will be moved to `DataViewSchemaExtensions` and `DataViewTypeExtensions`, 2 new extension classes that provide the following methods:

```cs
public static class DataViewSchemaExtensions
{
public static IEnumerable ToDataFrameColumns(this DataViewSchema schema, bool includesHidden = false, ISet? selectColumns = null);
}

public static class DataViewTypeExtensions
{
public static DataFrameColumn CreateDataFrameColumn(this DataViewType type, string columnName);
}
```

Note that `ToDataFrameColumns` also provides an option to include hidden column in the resulting `IEnumerable`.

**Describe alternatives you've considered**
Another way to solve this issue is to create an `IDataView` from `DataViewSchema` then convert the `IDataView` to `DataFrame` using the existing extension method. Unfortunately, [`EmptyDataView`](https://github.com/dotnet/machinelearning/blob/main/src/Microsoft.ML.Data/DataView/EmptyDataView.cs) is not `public`, and this involves some unnecessary operations.

**Additional context**
If the row data of the `DataFrame` are known, creating `DataFrameColumn` with the row data as values using [`Create`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.data.analysis.dataframecolumn.create?view=ml-dotnet-preview) likely has better performance than `DataViewSchemaExtensions.ToDataFrameColumns` followed by multiple `Append` calls. However, manual handling of `DataViewType` will be required.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.