tensorflow / tensorflow/models

@tensorflow-models/handpose running too slow on React Native

Open
#9,652 7 comments 0 reactions 1 assignee View on GitHub

@jaeyounkim is already working on this.

Since Jan 25, 2021.

models:research type:support
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

Hi everyone.
I'm trying to develop an app to process sign language using React Native, Expo and @mediapipe/handpose, but the performance I get is too slow for what we need. The smartphone I'm testing with is the Samsung Galaxy S9, which is not at all a bad phone, and the FPS I get is around 2 ~ 6 fps;
The process is very simple: I load the handpose model -> Then with the CameraWithTensors from TensorFlow Js React Native I can create a loop to process the frames from the smartphone camera -> Inside this loop I'm able to call the model.estimateHands();

But when the estimatehands method is called, even not rendering the camera, the performance drops absurdly, making it almost useless for what we want because we need to apply another model to process the points generated; from the examples I've seen at the mediapipe website, the results I'm getting are very uncongruent. Do you think the problem is being caused by my code?
Here's a video on how it's looking like:

https://user-images.githubusercontent.com/33132257/104114231-c2d7b300-52e0-11eb-8f5f-294029ede81d.mp4

Here's some sample code:
20210110_013138.jpg
20210110_013402.jpg
20210110_013304.jpg

(Sending photos because, as you can see at the edit history I couldn't manage to format it properly 🤭)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.