dotnet / dotnet/machinelearning

`ConvertToOnnx` should also accept `DataViewSchema`

Open
#6,448 6 comments 0 reactions 0 assignees View on GitHub
area-ONNX enhancement in-pr
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**Is your feature request related to a problem? Please describe.**
Currently, [saving a model to zip file](https://learn.microsoft.com/en-us/dotnet/api/microsoft.ml.modeloperationscatalog.save?view=ml-dotnet) only requires a `DataViewSchema`, but [saving a model to ONNX](https://learn.microsoft.com/en-us/dotnet/api/microsoft.ml.onnxexportextensions.converttoonnx?view=ml-dotnet-preview) requires `IDataView`.

Inside [`ConvertToOnnxProtobufCore`](https://github.com/dotnet/machinelearning/blob/main/src/Microsoft.ML.OnnxConverter/OnnxExportExtensions.cs#L18), a prediction was performed (`transform.Transform(inputData)`), which may be expensive if the training data set is large.

**Describe the solution you'd like**
`ConvertToOnnx` should have overloads that accept `DataViewSchema`, then convert the `DataViewSchema` to an [empty `IDataView`](https://github.com/dotnet/machinelearning/blob/main/src/Microsoft.ML.Data/DataView/EmptyDataView.cs), and pass the empty `IDataView` to the methods accepting `IDataView`.

The performance of methods accepting `IDataView` may be improved if `EmptyDataView` is created from the [Schema of the IDataView](https://learn.microsoft.com/en-us/dotnet/api/microsoft.ml.idataview.schema?view=ml-dotnet) and passed to `ConvertToOnnxProtobuf`, instead of full data.

**Describe alternatives you've considered**
Nil

**Additional context**
I have implemented the proposed solution and that seems working well. It is unfortunate that [`EmptyDataView`](https://github.com/dotnet/machinelearning/blob/main/src/Microsoft.ML.Data/DataView/EmptyDataView.cs) is an `internal` class, so I have to implement my own `EmptyDataView`.

```cs
sealed record EmptyDataView(DataViewSchema Schema) : IDataView {
public bool CanShuffle => true;

public long? GetRowCount() => 0L;

public DataViewRowCursor GetRowCursor(IEnumerable columnsNeeded, Random? rand = null)
=> new EmptyDataViewRowCursor(Schema);

public DataViewRowCursor[] GetRowCursorSet(IEnumerable columnsNeeded, int n, Random? rand = null)
=> Array.Empty();
}

sealed class EmptyDataViewRowCursor : DataViewRowCursor {
private readonly DataViewSchema schema;

public EmptyDataViewRowCursor(DataViewSchema Schema) {
schema = Schema;
}

public override DataViewSchema Schema => schema;

public override long Position => -1L;

public override bool IsColumnActive(DataViewSchema.Column column) => false;

public override bool MoveNext() => false;

public override long Batch => 0L;

public override ValueGetter GetGetter(DataViewSchema.Column column)
=> throw new InvalidOperationException();

public override ValueGetter GetIdGetter()
=> throw new InvalidOperationException();
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.