justadudewhohacks / justadudewhohacks/opencv4nodejs
Strange results w/ OpenPose (hands pose) and opencv4nodejs
- Dominant language
- C++
- Stars
- 5.1k
- Forks
- 826
- PR merge metrics
- No merged PRs in 30d
Description
We're trying to run source code in Python / C ++ / opencv translated into NodeJS / opencv4nodejs.
We have posted the code below.
Our interest is only related to hand poses. Openposes returns a Heatmap with 22 hand points in various hand poses.
Note that net.forward() returns a mat with the following sizes (1,22,46,46). But by calling mat.getDataAsArray(), it returns a three-dimensional array with [1,22,46] - is that correct?
The real result we obtained is wrong - shows some points at random positions in img. We are unable to interpret the heatmap returned by net.forward().
Original source at this link:
https://github.com/opencv/opencv/blob/master/samples/dnn/openpose.cpp
and
https://github.com/opencv/opencv/blob/master/samples/dnn/openpose.py
Our source:
```
import cv from 'opencv4nodejs';
import fs from 'fs';
import path from 'path';
function handDetect1(img) {
let POSE_PAIRS = [
[0,1],[1,2],[2,3],[3,4], // thumb
[0,5],[5,6],[6,7],[7,8], // pinkie
[0,9],[9,10],[10,11],[11,12], /// middle
[0,13],[13,14],[14,15],[15,16], // ring
[0,17],[17,18],[18,19],[19,20] // smal
];
let nPoints = 22;
const [frameHeight, frameWidth] = img.sizes;
let threshold = 0.01;
let inHeight = 368;
let inWidth = 368;
const inputBlob = cv.blobFromImage(img, 1 / 255.0, new cv.Size(inWidth, inHeight),
new cv.Vec(0,0,0), false, false);
const ssdcocoModelPath = './data/dnn/handpose';
if (!cv.modules.dnn) {
throw new Error('exiting: opencv4nodejs compiled without dnn module');
}
const prototxt = path.resolve(ssdcocoModelPath, 'pose_deploy.prototxt');
const modelFile = path.resolve(ssdcocoModelPath, 'pose_iter_102000.caffemodel');
if (!fs.existsSync(prototxt) || !fs.existsSync(modelFile)) {
throw new Error('exiting: could not find ssd pesrson pose net model');
}
let net = cv.readNetFromCaffe(prototxt, modelFile);
net.setInput(inputBlob);
let output = net.forward();
/*
output.sizes[0] == 1;
output.sizes[1] == 22;
output.sizes[2] == 46;
output.sizes[3] == 46;
*/
let H = output.sizes[2];
let W = output.sizes[3];
let SX = img.cols / W;
let SY = img.rows / H;
let outArray = output.getDataAsArray();
let points = [];
for(let n=0;n threshold) {
let point = maxLoc;
points.push(point);
img.drawCircle(point, 8, new cv.Vec(0,255,255));
img.putText(String(n), point, cv.FONT_HERSHEY_SIMPLEX, 1, new cv.Vec(0,0,255), 2);
}
else {
points.push(null);
}
}
return img;
}
```
Contributor guide
Research direction
Start with the linked OpenCV openpose.cpp and openpose.py examples, then inspect the handDetect1 path around net.forward() and getDataAsArray(). Compare the reported output shape and point coordinates with those examples; done means the returned heatmap dimensions are interpreted consistently and detected hand points are no longer random.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- computer-vision
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100