justadudewhohacks / justadudewhohacks/opencv4nodejs

Strange results w/ OpenPose (hands pose) and opencv4nodejs

Open
#652 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
5.1k
Forks
826
PR merge metrics
No merged PRs in 30d

Description

We're trying to run source code in Python / C ++ / opencv translated into NodeJS / opencv4nodejs.

We have posted the code below.

Our interest is only related to hand poses. Openposes returns a Heatmap with 22 hand points in various hand poses.

Note that net.forward() returns a mat with the following sizes (1,22,46,46). But by calling mat.getDataAsArray(), it returns a three-dimensional array with [1,22,46] - is that correct?

The real result we obtained is wrong - shows some points at random positions in img. We are unable to interpret the heatmap returned by net.forward().

Original source at this link:

https://github.com/opencv/opencv/blob/master/samples/dnn/openpose.cpp

and

https://github.com/opencv/opencv/blob/master/samples/dnn/openpose.py

Our source:

```

import cv from 'opencv4nodejs';
import fs from 'fs';
import path from 'path';

function handDetect1(img) {
let POSE_PAIRS = [
[0,1],[1,2],[2,3],[3,4], // thumb
[0,5],[5,6],[6,7],[7,8], // pinkie
[0,9],[9,10],[10,11],[11,12], /// middle
[0,13],[13,14],[14,15],[15,16], // ring
[0,17],[17,18],[18,19],[19,20] // smal
];
let nPoints = 22;
const [frameHeight, frameWidth] = img.sizes;
let threshold = 0.01;
let inHeight = 368;
let inWidth = 368;
const inputBlob = cv.blobFromImage(img, 1 / 255.0, new cv.Size(inWidth, inHeight),
new cv.Vec(0,0,0), false, false);

const ssdcocoModelPath = './data/dnn/handpose';
if (!cv.modules.dnn) {
throw new Error('exiting: opencv4nodejs compiled without dnn module');
}
const prototxt = path.resolve(ssdcocoModelPath, 'pose_deploy.prototxt');
const modelFile = path.resolve(ssdcocoModelPath, 'pose_iter_102000.caffemodel');

if (!fs.existsSync(prototxt) || !fs.existsSync(modelFile)) {
throw new Error('exiting: could not find ssd pesrson pose net model');
}

let net = cv.readNetFromCaffe(prototxt, modelFile);

net.setInput(inputBlob);

let output = net.forward();
/*
output.sizes[0] == 1;
output.sizes[1] == 22;
output.sizes[2] == 46;
output.sizes[3] == 46;
*/
let H = output.sizes[2];
let W = output.sizes[3];
let SX = img.cols / W;
let SY = img.rows / H;
let outArray = output.getDataAsArray();

let points = [];
for(let n=0;n threshold) {
let point = maxLoc;
points.push(point);

img.drawCircle(point, 8, new cv.Vec(0,255,255));
img.putText(String(n), point, cv.FONT_HERSHEY_SIMPLEX, 1, new cv.Vec(0,0,255), 2);
}
else {
points.push(null);
}
}

return img;
}

```

Contributor guide

Open the contributing guide

Research direction

Start with the linked OpenCV openpose.cpp and openpose.py examples, then inspect the handDetect1 path around net.forward() and getDataAsArray(). Compare the reported output shape and point coordinates with those examples; done means the returned heatmap dimensions are interpreted consistently and detected hand points are no longer random.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
computer-vision
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.