Larq Compute Engine seems incompatible with tensorflow-lite-task-vision on Android (using the latest tensorflow lite demo code)
Open
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 258
- Forks
- 35
- PR merge metrics
- No merged PRs in 30d
Description
I would like to build an Android app using the Larq Compute Engine.
I started by following the Android tutorial of the Larq website, however I had to adapt the tutorial quite a bit since the tensorflow lite demo project that is referenced changed quite a bit since the moment the tutorial was written in the Larq documentation.
I added the quicknet.tflite model from the Larq zoo to the project, but I am getting the following error:
2024-01-26 14:17:42.513 11652-11766 helper org....examples.imageclassification I using model quicknet.tflite
2024-01-26 14:17:42.515 11652-11766 tflite org....examples.imageclassification I Created TensorFlow Lite XNNPACK delegate for CPU.
2024-01-26 14:17:42.515 11652-11766 tflite org....examples.imageclassification E Encountered unresolved custom op: LceQuantize.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom
2024-01-26 14:17:42.515 11652-11766 tflite org....examples.imageclassification E Node number 3 (LceQuantize) failed to prepare.
2024-01-26 14:17:42.515 11652-11766 tflite org....examples.imageclassification E Encountered unresolved custom op: LceQuantize.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom
2024-01-26 14:17:42.515 11652-11766 tflite org....examples.imageclassification E Node number 3 (LceQuantize) failed to prepare.
2024-01-26 14:17:42.517 11652-11766 TaskJniUtils org....examples.imageclassification E Error getting native address of native library: task_vision_jni
java.lang.IllegalArgumentException: Error occurred when initializing ImageClassifier: Encountered unresolved custom op: LceQuantize.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom
at org.tensorflow.lite.task.vision.classifier.ImageClassifier.initJniWithModelFdAndOptions(Native Method)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier.access$000(ImageClassifier.java:73)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:142)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:135)
at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier.createFromFileAndOptions(ImageClassifier.java:133)
at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.setupImageClassifier(ImageClassifierHelper.java:172)
at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.classify(ImageClassifierHelper.java:187)
at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.classifyImage(CameraFragment.java:349)
at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.lambda$bindCameraUseCases$7$org-tensorflow-lite-examples-imageclassification-fragments-CameraFragment(CameraFragment.java:315)
at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment$$ExternalSyntheticLambda10.analyze(Unknown Source:2)
at androidx.camera.core.ImageAnalysis.lambda$setAnalyzer$2(ImageAnalysis.java:481)
at androidx.camera.core.ImageAnalysis$$ExternalSyntheticLambda2.analyze(Unknown Source:2)
at androidx.camera.core.ImageAnalysisAbstractAnalyzer.lambda$analyzeImage$0$androidx-camera-core-ImageAnalysisAbstractAnalyzer(ImageAnalysisAbstractAnalyzer.java:285)
at androidx.camera.core.ImageAnalysisAbstractAnalyzer$$ExternalSyntheticLambda1.run(Unknown Source:14)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:923)
2024-01-26 14:17:42.521 11652-11766 AndroidRuntime org....examples.imageclassification E FATAL EXCEPTION: pool-2-thread-1
Process: org.tensorflow.lite.examples.imageclassification, PID: 11652
java.lang.IllegalStateException: Error getting native address of native library: task_vision_jni
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:95)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier.createFromFileAndOptions(ImageClassifier.java:133)
at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.setupImageClassifier(ImageClassifierHelper.java:172)
at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.classify(ImageClassifierHelper.java:187)
at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.classifyImage(CameraFragment.java:349)
at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.lambda$bindCameraUseCases$7$org-tensorflow-lite-examples-imageclassification-fragments-CameraFragment(CameraFragment.java:315)
at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment$$ExternalSyntheticLambda10.analyze(Unknown Source:2)
at androidx.camera.core.ImageAnalysis.lambda$setAnalyzer$2(ImageAnalysis.java:481)
at androidx.camera.core.ImageAnalysis$$ExternalSyntheticLambda2.analyze(Unknown Source:2)
at androidx.camera.core.ImageAnalysisAbstractAnalyzer.lambda$analyzeImage$0$androidx-camera-core-ImageAnalysisAbstractAnalyzer(ImageAnalysisAbstractAnalyzer.java:285)
at androidx.camera.core.ImageAnalysisAbstractAnalyzer$$ExternalSyntheticLambda1.run(Unknown Source:14)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:923)
Caused by: java.lang.IllegalArgumentException: Error occurred when initializing ImageClassifier: Encountered unresolved custom op: LceQuantize.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom
at org.tensorflow.lite.task.vision.classifier.ImageClassifier.initJniWithModelFdAndOptions(Native Method)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier.access$000(ImageClassifier.java:73)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:142)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:135)
at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
... 14 more
2024-01-26 14:17:42.522 11652-11770 helper org....examples.imageclassification I using model quicknet.tflite
2024-01-26 14:17:42.523 11652-11770 tflite org....examples.imageclassification E Encountered unresolved custom op: LceQuantize.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom
2024-01-26 14:17:42.523 11652-11770 tflite org....examples.imageclassification E Node number 3 (LceQuantize) failed to prepare.
2024-01-26 14:17:42.523 11652-11770 tflite org....examples.imageclassification E Encountered unresolved custom op: LceQuantize.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom
2024-01-26 14:17:42.523 11652-11770 tflite org....examples.imageclassification E Node number 3 (LceQuantize) failed to prepare.
2024-01-26 14:17:42.527 11652-11770 TaskJniUtils org....examples.imageclassification E Error getting native address of native library: task_vision_jni
java.lang.IllegalArgumentException: Error occurred when initializing ImageClassifier: Encountered unresolved custom op: LceQuantize.
See instructions: https://www.tensorflow.org/lite/guide/ops_custom
at org.tensorflow.lite.task.vision.classifier.ImageClassifier.initJniWithModelFdAndOptions(Native Method)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier.access$000(ImageClassifier.java:73)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:142)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:135)
at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)
at org.tensorflow.lite.task.vision.classifier.ImageClassifier.createFromFileAndOptions(ImageClassifier.java:133)
at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.setupImageClassifier(ImageClassifierHelper.java:172)
at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.classify(ImageClassifierHelper.java:187)
at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.classifyImage(CameraFragment.java:349)
at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.lambda$bindCameraUseCases$7$org-tensorflow-lite-examples-imageclassification-fragments-CameraFragment(CameraFragment.java:315)
at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment$$ExternalSyntheticLambda10.analyze(Unknown Source:2)
at androidx.camera.core.ImageAnalysis.lambda$setAnalyzer$2(ImageAnalysis.java:481)
at androidx.camera.core.ImageAnalysis$$ExternalSyntheticLambda2.analyze(Unknown Source:2)
at androidx.camera.core.ImageAnalysisAbstractAnalyzer.lambda$analyzeImage$0$androidx-camera-core-ImageAnalysisAbstractAnalyzer(ImageAnalysisAbstractAnalyzer.java:285)
at androidx.camera.core.ImageAnalysisAbstractAnalyzer$$ExternalSyntheticLambda1.run(Unknown Source:14)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:923)
Second test
I tried to make another blank project just to see if I could execute an inference with a binary model using Larq at all, and I succeeded by only using the following dependencies in this other project:
implementation 'org.larq:lce-lite:0.1.000'
implementation 'org.tensorflow:tensorflow-lite:2.14.0'
implementation 'org.tensorflow:tensorflow-lite-gpu:2.14.0'
I tried other versions of the tflite packages like 1.14.0 and 2.8.0 but it throwed JNI errors when creating the Interpreter object.
Returning to the tensorflow demo project and the Larq tutorial
The tensorflow demo uses the following dependencies:
implementation 'org.tensorflow:tensorflow-lite:2.14.0'
implementation 'org.tensorflow:tensorflow-lite-gpu:2.14.0'
// Tensorflow lite dependencies
implementation 'org.tensorflow:tensorflow-lite-support:0.4.0'
implementation 'org.tensorflow:tensorflow-lite-task-vision:0.4.0'
// Import the GPU delegate plugin Library for GPU inference
implementation 'org.tensorflow:tensorflow-lite-gpu:2.9.0'
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin:0.4.0'
Note that I am not excluding the tensorflow-lite module like in the tutorial, I am using the following instead:
packagingOptions {
pickFirst 'lib/arm64-v8a/libtensorflowlite_jni.so'
pickFirst 'lib/x86/libtensorflowlite_jni.so'
pickFirst 'lib/x86_64/libtensorflowlite_jni.so'
}
My uneducated guess is that the issue is caused by tensorflow-lite-task-vision which seems to include its own native code libraries under other names, so that may explain the line Error getting native address of native library: task_vision_jni.
Conclusion
I guess either the tutorial can be updated to take into account the changes in the tensorflow demo repository, if failed to adapt the code correctly, or the Larq compute engine library needs to be changed to be compatible with the additional tflite modules they use in the official tflite demos.
Also, I am wondering if disabling XNNPack is still necessary? I couldn't find any way to disable it with the newest versions of tflite, but it seems to work fine with default options in my second test, I mean there is no error or warning at least.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Larq Android tutorial and the referenced TensorFlow Lite demo project, then inspect ImageClassifierHelper.java and CameraFragment.java around the classifier setup and failure. Reproduce the unresolved LceQuantize error with quicknet.tflite; done means the model can initialize through tensorflow-lite-task-vision without the reported crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, cpp
- Domain
- machine-learning, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100