googlesamples / googlesamples/mlkit

[Feature request] Using ArImage as InputImage parameter

Open
#875 0 comments 0 reactions 1 assignee Claimed by @jackqdyulei View on GitHub
Dominant language
Kotlin
Stars
4.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

**What's your feature request? Please describe.**

In my current project, ARCore is fully implemented in native code (C/C++). The captured camera CPU buffer is of type `ArImage`, which is essentially same as `InputImage.IMAGE_FORMAT_YUV_420_888` format. I'm providing the YUV plane data from the native layer to the Java side through JNI, and passing the camera frame to MLKit as an `InputImage` in Java. However, while MLKit’s does offer `InputImage.IMAGE_FORMAT_YUV_420_888` along with `InputImage.fromMediaImage`, the image data must be provided in the form of `android.media.Image`.

Unfortunately, there doesn't seem to be a way to instantiate native C/C++ YUV plane buffers as `android.media.Image`. I currently convert the data to a format like NV21 and then pass it to MLKit using `InputImage.fromByteArray` with `InputImage.IMAGE_FORMAT_NV21` (which works well smoothly).

However, I feel like this additional process is still no needed. If I were using ARCore in Java and utilized [acquireCameraImage](https://developers.google.com/ar/reference/java/com/google/ar/core/Frame#acquireCameraImage()), the data would be provided as `android.media.Image` from the start, so this issue wouldn't arise. However, in my current project, ARCore Java cannot be used, so I need to use [arframe_acquirecameraimage](https://developers.google.com/ar/reference/c/group/ar-frame#arframe_acquirecameraimage) instead.

As mentioned in the issue #481, while I understand that providing MLKit as NDK would take long anyway, it would be helpful if the only `InputImage` could support YUV plane data directly. Or if there is another good approach that I might be unaware of and this feature addition is unnecessary, I would greatly appreciate it if you could let me know as well. Thank you. :)

**Mobile environment**
Android

**Additional context**
None

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.