dotnet / dotnet/machinelearning
Easier to create `DataFrame` from `DataViewSchema`
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
**Is your feature request related to a problem? Please describe.**
[Loading a model](https://learn.microsoft.com/en-us/dotnet/api/microsoft.ml.modeloperationscatalog.load?view=ml-dotnet) returns a `ITransformer` and its input column information in a `DataViewSchema`. `IDataView` is required for making predictions and creating a `DataFrame` should be the simplest way, especially when it provides [`Append`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.data.analysis.dataframe.append?view=ml-dotnet-preview) to add data row by row. Unfortunately, creating a `DataFrame` from a `DataViewSchema` is currently not easy because the `DataFrame` constructors basically only accepts `IEnumerable` and there is no direct conversion from `DataViewSchema` to `IEnumerable`.
**Describe the solution you'd like**
`IDataViewExtensions.ToDataFrame` actually contains the logic converting `DataViewSchema` to `IEnumerable`. To solve this issue, that part of the code will be moved to `DataViewSchemaExtensions` and `DataViewTypeExtensions`, 2 new extension classes that provide the following methods:
```cs
public static class DataViewSchemaExtensions
{
public static IEnumerable ToDataFrameColumns(this DataViewSchema schema, bool includesHidden = false, ISet? selectColumns = null);
}
public static class DataViewTypeExtensions
{
public static DataFrameColumn CreateDataFrameColumn(this DataViewType type, string columnName);
}
```
Note that `ToDataFrameColumns` also provides an option to include hidden column in the resulting `IEnumerable`.
**Describe alternatives you've considered**
Another way to solve this issue is to create an `IDataView` from `DataViewSchema` then convert the `IDataView` to `DataFrame` using the existing extension method. Unfortunately, [`EmptyDataView`](https://github.com/dotnet/machinelearning/blob/main/src/Microsoft.ML.Data/DataView/EmptyDataView.cs) is not `public`, and this involves some unnecessary operations.
**Additional context**
If the row data of the `DataFrame` are known, creating `DataFrameColumn` with the row data as values using [`Create`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.data.analysis.dataframecolumn.create?view=ml-dotnet-preview) likely has better performance than `DataViewSchemaExtensions.ToDataFrameColumns` followed by multiple `Append` calls. However, manual handling of `DataViewType` will be required.
Contributor guide
Assessment
This issue has not been assessed yet.