margelo / margelo/react-native-fast-tflite
Output values are not changing for different inputs
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1.2k
- Forks
- 92
- Avg merge
- 13h 9m
- Merged PRs (30d)
- 2
Description
Hi, I am using a movenet model from tfhub.dev with FrameProcessor and VisionCamera to try and apply human pose estimation to a person. It doesn't appear as though it is tracking my movements as the outputs in the console are always the same. This appears to be the case with all models I try to use.
Here is the code I am using to resize the frame:
function getArrayFromCache(size) {
'worklet'
if (global[CACHE_ID] == null || global[CACHE_ID].length !== size) {
global[CACHE_ID] = new Uint8Array(size);
}
return global[CACHE_ID];
}
function resize(frame, width, height) {
'worklet'
const inputWidth = frame.width;
const inputHeight = frame.height;
const arrayData = frame.toArrayBuffer();
const outputSize = width * height * 3; // 3 for RGB
const outputFrame = getArrayFromCache(outputSize);
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
// Find closest pixel from the source image
const srcX = Math.floor((x / width) * inputWidth);
const srcY = Math.floor((y / height) * inputHeight);
// Compute the source and destination index
const srcIndex = (srcY * inputWidth + srcX) * 4; // 4 for BGRA
const destIndex = (y * width + x) * 3; // 3 for RGB
// Convert from BGRA to RGB
outputFrame[destIndex] = arrayData[srcIndex + 2]; // R
outputFrame[destIndex + 1] = arrayData[srcIndex + 1]; // G
outputFrame[destIndex + 2] = arrayData[srcIndex]; // B
}
}
return outputFrame;
}
Here is my frame processor function:
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
if (model == null) return
const newFrame = resize(frame, 192, 192)
const outputs = model.runSync([newFrame])
outputs = outputs[0]
console.log(outputs[1])
}, [model])
Here is the output in the console:
LOG 0.46377456188201904
LOG 0.46377456188201904
LOG 0.46377456188201904
LOG 0.46377456188201904
LOG 0.46377456188201904
LOG 0.46377456188201904
For each frame the camera sees the result is always the same.
Does anyone know how to resolve this issue?
Thank you
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the resize function and frameProcessor shown in the issue, then check the model's expected input format and the frame data passed to runSync. Verify whether successive frames contain different pixel values before investigating model outputs. Done means identifying why inputs or outputs remain constant and documenting or testing the fix; no repository file or test is named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- react-native, tensorflow, typescript
- Domain
- machine-learning, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100