googlesamples / googlesamples/mlkit
[Bug report] InputImage.fromMediaImage not applying rotation
- Dominant language
- Kotlin
- Stars
- 4.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
InputImage.fromMediaImage and InputImage.fromBitmap do not seem to be applying parameter rotationDegrees properly, at least when used with text-recognition.
I'm attempting to fix MLKit results when used with `react-native-vision-camera`, and discovered that all `rotationDegrees` values supplied as input result in the exact same detection outcomes as not rotating the image at all (180 degrees rotation from upright sensor returns text lines in the opposite order). The data is YUV_420_888 directly from Camera2/CameraX as `Image`.
Converting the YUV to Bitmap manually and using `InputImage.fromBitmap` returned the same results.
The only way I could circumvent this was by rotating the bitmap before creating the InputImage, or setting automatic rotation of the buffers in CameraX.
My original issue in `vision-camera` https://github.com/mrousavy/react-native-vision-camera/issues/3490
**To Reproduce**
Take Image from CameraX or Camera 2 when rotated to 180 degrees from upright sensor, process with InputImage, and process with TextRecognizer. Check the order of the lines in returned text.
Getting the sample app up and running and modifying all relevant parts to reproduce this is a huge task for me, but here's the callback logic I have tested with vision-camera
```kt
private var recognizer = TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS)
override fun callback(frame: Frame, arguments: Map?): HashMap? {
val data = WritableNativeMap()
val mediaImage: Image = frame.image
val image =
InputImage.fromMediaImage(mediaImage, 180) // Hard coded for testing purposes, no values worked as expected
val task: Task = recognizer.process(image)
try {
val text: Text = Tasks.await(task)
if (text.text.isEmpty()) {
return WritableNativeMap().toHashMap()
}
data.putString("resultText", text.text) // This text comes in reversed order when sensor is at 180 deg
data.putArray("blocks", getBlocks(text.textBlocks))
return data.toHashMap()
} catch (e: Exception) {
e.printStackTrace()
return null
}
}
```
**Expected behavior**
Expect resulting MediaImage to be rotated and result in expected Text Recognition output ordering.
**SDK Info:**
- com.google.android.gms:play-services-mlkit-text-recognition:19.0.1
- com.google.android.gms:play-services-mlkit-text-recognition-chinese:16.0.1 (+ others at same version)
- com.google.mlkit:translate:17.0.3
Also tested with latest non-play services variants.
**Smartphone:**
- Device/Simulator: Pixel 9 Pro, Asus Zenfone 8
- Device/Simulator OS: Android API 33, Android API 35
**Development Environment:**
*(For Android issue feel free to skip this section)*
- IDE Eversion: Android Studio Meerkat | 2024.3.1 Patch 1
- Laptop/Desktop: MacBook Pro M3 Max
- Laptop/Desktop OS/version: MacOS 15.4
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.