dotnet / dotnet/machinelearning

ImagePixelExtractingEstimator should not require the ImageResizingEstimator in the pipeline

Open
#2,418 3 comments 0 reactions 0 assignees View on GitHub
api-suggestion area-Vision enhancement Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

Currently in ML.Net, to be able to use an ImagePixelExtractingTransformer/ImagePixelExtractingEstimator, you need to have an ImageResizingEstimator in the pipeline before the ImagePixelExtracting, otherwise the pipeline [fails at run-time ](https://github.com/dotnet/machinelearning/blob/master/src/Microsoft.ML.ImageAnalytics/ImagePixelExtractorTransform.cs#L447)with a schema mismatch error.

`"'Schema mismatch for input column 'ImageObject': expected known-size image, got unknown-size image'". `

This happen because the ImagePixelExtracting operating on [ImageTypes ](https://github.com/dotnet/machinelearning/blob/master/src/Microsoft.ML.ImageAnalytics/ImageType.cs)needs the Width and the Height properties, to extract the pixels.

The Estimator that creates the [ImageType ](https://github.com/dotnet/machinelearning/blob/master/src/Microsoft.ML.ImageAnalytics/ImageType.cs)object, is the ImageLoadingEstimator, but the Width and Height properties of the estimator don't get filled until they get passed to ImageResizing.

Proposed fix:
- The ImageLoadingEstimator should allow the user to specify the height and width of the images, if the user has images of fixed with and height, rather than fail at "ExtractPixels" if resize is not present.
- if we make a bool about `imagesHaveSameDimentions`, and that is set to `yes`, the ImageLoadingTransformer could memorize the width and height of the first image, and check every subsequent image, drop all the samples that don't have the right dimensions, and warn about the records dropped.

cc @yaeldekel @Ivanidzo4ka @TomFinley

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.