dotnet / dotnet/machinelearning
[Feature Request] Create MLImage from SKBitmap instance in ML
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
**Is your feature request related to a problem? Please describe.**
Consider the following code snippet
```csharp
MLContext mlContext = new MLContext()
{
FallbackToCpu = true,
GpuDeviceId = 0
};
var pipeline = mlContext.Transforms.ResizeImages(inputColumnName: "bitmap", outputColumnName: "input_1",
imageWidth: 416, imageHeight: 416, resizing: ResizingKind.IsoPad)
.Append(mlContext.Transforms.ExtractPixels(outputColumnName: "input_1", scaleImage: 1f / 255f))
.Append(mlContext.Transforms.Concatenate("image_shape", "height", "width"))
.Append(mlContext.Transforms.ApplyOnnxModel(
shapeDictionary: new Dictionary() { { "input_1", new[] { 1, 3, 416, 416 } } },
inputColumnNames: new[]
{
"input_1",
"image_shape"
},
outputColumnNames: new[]
{
"yolonms_layer_1/ExpandDims_1:0",
"yolonms_layer_1/ExpandDims_3:0",
"yolonms_layer_1/concat_2:0"
},
modelFile: modelPath, recursionLimit: 100));
var model = pipeline.Fit(mlContext.Data.LoadFromEnumerable(new List()));
var predictionEngine = mlContext.Model.CreatePredictionEngine(model);
using var bitmap = SKBitmap.Decode("C:\test.jpg");
var predict = predictionEngine.Predict(new YoloV3BitmapData()
{ Image = MLImage.CreateFromFile("C:\test.jpg") });
var results = GetResults(predict, classesNames);
using var canvas = new SKCanvas(bitmap);
var paint = new SKPaint()
{
Color = new SKColor(Color.Red.R, 0, 0,50),
Style = SKPaintStyle.Fill
};
foreach (var result in results)
{
var y1 = result.BBox[0];
var x1 = result.BBox[1];
var y2 = result.BBox[2];
var x2 = result.BBox[3];
canvas.DrawRect(x1, y1, x2 - x1, y2 - y1, paint);
var paint2 = new SKPaint()
{
Color = new SKColor(0xff, 00, 00),
Style = SKPaintStyle.Fill,
Typeface = SKTypeface.FromFamilyName(
"Arial",
SKFontStyleWeight.Bold,
SKFontStyleWidth.Normal,
SKFontStyleSlant.Italic),
TextSize = 18,
IsAntialias = true,
TextAlign = SKTextAlign.Center,
IsStroke = false
};
canvas.DrawText(result.Label + " " + result.Confidence.ToString("0.00"), new SKPoint(x1, y1),
paint2);
}
public class YoloV3BitmapData
{
[ColumnName("bitmap")]
[ImageType(416, 416)]
public MLImage Image { get; set; }
[ColumnName("width")]
public float ImageWidth => Image.Width;
[ColumnName("height")]
public float ImageHeight => Image.Height;
}
```
In the above code, we use ``SKBitmap.Decode`` and ``MLImage.CreateFromFile`` to load the image from the file, which results in 2 IO,
So if you can directly pass SKBitmap to MLImage, you will save 1 IO.
So we can write
```csharp
using var bitmap = SKBitmap.Decode("C:\test.jpg");
var predict = predictionEngine.Predict(new YoloV3BitmapData() { Image = bitmap });
```
or
```csharp
using var bitmap = SKBitmap.Decode("C:\test.jpg");
var predict = predictionEngine.Predict(new YoloV3BitmapData() { Image = MLImage.CreateFromSKBitmap(bitmap)});
```
**Describe the solution you'd like**
We want to be able to simply pass SKBitmap to MLImage.
**Describe alternatives you've considered**
Currently it is possible to do something similar with the following code, but it is more redundant and error-prone
```csharp
using var fs=File.OpenRead("C:\test.jpg");
using var bitmap = SKBitmap.Decode(fs);
var predict = predictionEngine.Predict(new YoloV3BitmapData() { Image = MLImage.CreateFromStream(fs) });
```
**Additional context**
In addition, it would be useful to be able to expose the SKBitmap instance in MLImage, because this allows us to draw the detection results on the SKBitmap after obtaining the model detection results without additional coding.
Contributor guide
Assessment
This issue has not been assessed yet.