margelo / margelo/react-native-fast-tflite

Output values are not changing for different inputs

Open
#64 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
1.2k
Forks
92
Avg merge
13h 9m
Merged PRs (30d)
2

Description

Hi, I am using a movenet model from tfhub.dev with FrameProcessor and VisionCamera to try and apply human pose estimation to a person. It doesn't appear as though it is tracking my movements as the outputs in the console are always the same. This appears to be the case with all models I try to use.

Here is the link to the model

Here is the code I am using to resize the frame:

function getArrayFromCache(size) {
    'worklet'
    if (global[CACHE_ID] == null || global[CACHE_ID].length !== size) {
      global[CACHE_ID] = new Uint8Array(size);
    }
    return global[CACHE_ID];
  }

function resize(frame, width, height) {
    'worklet'
    const inputWidth = frame.width;
    const inputHeight = frame.height;
    const arrayData = frame.toArrayBuffer();

    const outputSize = width * height * 3; // 3 for RGB
    const outputFrame = getArrayFromCache(outputSize);

    for (let y = 0; y < height; y++) {
      for (let x = 0; x < width; x++) {
        // Find closest pixel from the source image
        const srcX = Math.floor((x / width) * inputWidth);
        const srcY = Math.floor((y / height) * inputHeight);

        // Compute the source and destination index
        const srcIndex = (srcY * inputWidth + srcX) * 4; // 4 for BGRA
        const destIndex = (y * width + x) * 3;           // 3 for RGB

        // Convert from BGRA to RGB
        outputFrame[destIndex] = arrayData[srcIndex + 2];   // R
        outputFrame[destIndex + 1] = arrayData[srcIndex + 1]; // G
        outputFrame[destIndex + 2] = arrayData[srcIndex];     // B
      }
    }

    return outputFrame;
  }

Here is my frame processor function:

  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    if (model == null) return

    const newFrame = resize(frame, 192, 192)

    const outputs = model.runSync([newFrame])
    outputs = outputs[0]
    console.log(outputs[1])
  }, [model])

Here is the output in the console:


 LOG  0.46377456188201904
 LOG  0.46377456188201904
 LOG  0.46377456188201904
 LOG  0.46377456188201904
 LOG  0.46377456188201904
 LOG  0.46377456188201904

For each frame the camera sees the result is always the same.

Does anyone know how to resolve this issue?

Thank you

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the resize function and frameProcessor shown in the issue, then check the model's expected input format and the frame data passed to runSync. Verify whether successive frames contain different pixel values before investigating model outputs. Done means identifying why inputs or outputs remain constant and documenting or testing the fix; no repository file or test is named.

Written by the indexing model from the issue text.

Assessment

Tech stack
react-native, tensorflow, typescript
Domain
machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.