infinitered / infinitered/react-native-mlkit
Stream camera frame to ObjectDetector?
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 444
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
I have been using React-Native-Vision-Camera + react-native-fast-tflite. Unfortunately, I need to investigate an alternate solution to passing images to the model. This happens seamlessly with RNFastTFLite. I noticed with this MLKIt library, that a `uri` path "string" is needed to pass to the model. That would require converting the `frame` from RNVisionCamera's `frameProcessor` to an image on the device and then create a temp file path. This is going to be very memory & storage intensive, so not a good idea.
Does this MLKIt work well with frame streams, or is it designed to be more for single image processing? I see you can set `detectorMode: 'stream'` option for when initializing the original model options, but it still requires a string path?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the ObjectDetector input path and the `detectorMode: 'stream'` option, then compare them with React-Native-Vision-Camera's `frameProcessor` output and RNFastTFLite's frame handling. Determine whether frames can be passed without creating temporary image files; done should be a documented answer or a clearly scoped implementation path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- react-native, typescript
- Domain
- computer-vision, mobile
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100