dotnet / dotnet/machinelearning

Improve flexibility of input types for image classification

Open
#6,590 2 comments 1 reaction 0 assignees View on GitHub
area-Vision enhancement needs-further-triage
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**Is your feature request related to a problem? Please describe.**
The image classifier in ML.NET currently accepts input in the form of a byte array (only certain file formats, e.g. png jpeg gif), which is flexible enough for many scenarios. However, it is sometimes the case that the source image originates in a System.Image.Bitmap, OpenCV mat, or other managed/unmanaged memory block not in png/jpeg/gif format. To perform a prediction with this sort of data, it must be converted which takes time and requires allocations for data that already exists.

In a production computer vision environment where image data is streamed in as System.Drawing.Bitmap objects whose RawFormat is MemoryBmp, it is undesirable to dedicate 25-30ms per image to save each Bitmap into a managed byte[] whose RawFormat is jpeg (80ms for png), if it can be avoided by an update to allow more direct consumption of other formats and datatypes.

**Describe the solution you'd like**
Direct consumption of System.Drawing.Bitmap (RawFormat=MemoryBmp) by ML.NET in a way which avoids managed allocations or format conversion would be ideal. Accepting an IntPtr to unmanaged pixel data would be fine as well, with support for bmp format.

**Describe alternatives you've considered**
Various methods to convert Bitmap data into byte[] were tested and are functional, but none achieves the performance desired.

**Additional context**
The general use case for this is for processing of images received via a camera which streams images in as Bitmap objects. Needing to convert this to managed memory in one of several formats is a bottleneck which would ideally be avoidable through the solution described above.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.