googlesamples / googlesamples/mlkit
[Feature request] Using ArImage as InputImage parameter
- Dominant language
- Kotlin
- Stars
- 4.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
**What's your feature request? Please describe.**
In my current project, ARCore is fully implemented in native code (C/C++). The captured camera CPU buffer is of type `ArImage`, which is essentially same as `InputImage.IMAGE_FORMAT_YUV_420_888` format. I'm providing the YUV plane data from the native layer to the Java side through JNI, and passing the camera frame to MLKit as an `InputImage` in Java. However, while MLKit’s does offer `InputImage.IMAGE_FORMAT_YUV_420_888` along with `InputImage.fromMediaImage`, the image data must be provided in the form of `android.media.Image`.
Unfortunately, there doesn't seem to be a way to instantiate native C/C++ YUV plane buffers as `android.media.Image`. I currently convert the data to a format like NV21 and then pass it to MLKit using `InputImage.fromByteArray` with `InputImage.IMAGE_FORMAT_NV21` (which works well smoothly).
However, I feel like this additional process is still no needed. If I were using ARCore in Java and utilized [acquireCameraImage](https://developers.google.com/ar/reference/java/com/google/ar/core/Frame#acquireCameraImage()), the data would be provided as `android.media.Image` from the start, so this issue wouldn't arise. However, in my current project, ARCore Java cannot be used, so I need to use [arframe_acquirecameraimage](https://developers.google.com/ar/reference/c/group/ar-frame#arframe_acquirecameraimage) instead.
As mentioned in the issue #481, while I understand that providing MLKit as NDK would take long anyway, it would be helpful if the only `InputImage` could support YUV plane data directly. Or if there is another good approach that I might be unaware of and this feature addition is unnecessary, I would greatly appreciate it if you could let me know as well. Thank you. :)
**Mobile environment**
Android
**Additional context**
None
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.