googlesamples / googlesamples/mlkit

[Bug report] Barcode AutoZoom not working with coordinate matrix in InputImage

Open
#844 0 comments 1 reaction 1 assignee Claimed by @zhouyiself View on GitHub
Dominant language
Kotlin
Stars
4.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

The auto-zoom feature of the barcode scanner does not work when using a `InputImage` that has a coordinate matrix attached.
In my code I use the matrix to automatically map the ml-kit barcode coordinates to my preview views coordinate system.

**To Reproduce**
1. Create a barcode scanner with configured auto-zoom
2. In the `ImageAnalysis` create the `InputImage` using a `Matrix` and pass the image to the `process` function
4. No auto-zoom will happen

The code to create the Matrix is the same as used in the [`MlKitAnalyzer`](https://developer.android.com/media/camera/camerax/mlkitanalyzer) and does successfully map the coordinates.

When not using a custom matrix I can see the following logcat output and the camera is zoomed in:

```
I Process PredictedArea: [0,58, 0,00, 0,72, 0,66, 0,00], frameIndex = 101
I Process PredictedArea: [0,59, 0,01, 0,75, 0,36, 0,00], frameIndex = 102
I Process PredictedArea: [0,52, 0,05, 0,74, 0,29, 0,00], frameIndex = 105
I Process PredictedArea: [0,52, 0,06, 0,74, 0,28, 0,00], frameIndex = 106
I Process PredictedArea: [0,52, 0,01, 0,75, 0,27, 0,00], frameIndex = 107
I Process PredictedArea: [0,48, 0,01, 0,74, 0,27, 0,00], frameIndex = 108
I Going to set zoom = 1.9689503
```

With my custom matrix the following is logged and no zooming is happening (Camera is about the same distance from the code as in the first example):

```
I Process PredictedArea: [0,60, 0,54, 0,69, 0,93, 0,00], frameIndex = 1
I Process PredictedArea: [0,57, 0,52, 0,69, 0,93, 0,00], frameIndex = 2
I Process PredictedArea: [0,58, 0,60, 0,69, 0,93, 0,00], frameIndex = 3
I Process PredictedArea: [0,59, 0,64, 0,70, 0,93, 0,00], frameIndex = 4
I Process PredictedArea: [0,58, 0,61, 0,70, 0,97, 0,00], frameIndex = 5
I Process PredictedArea: [0,58, 0,59, 0,70, 0,97, 0,00], frameIndex = 6
I Auto zoom to 1.0 is filtered by threshold
```

I suspect that this might be because when mapping the coordinates using the matrix, the size of the codes is increased and thus the AutoZoom algorithm thinks that a zoom is not needed.
To solve that problem, the AutoZoom should calculate the distance based on the unmodified bounding box of the barcode instead of with the mapped coordinates.

**Expected behavior**
The zoom ratio should be updated to zoom in on far away codes.

**SDK Info:**
- com.google.android.gms:play-services-mlkit-barcode-scanning:18.3.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.