JdeRobot / JdeRobot/VisualCircuit

New CV block proposal: MediaPipe Hand & Gesture detector

Open
#470 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
20
Forks
34
Avg merge
4h 56m
Merged PRs (30d)
10

Description

Hi @jmplaza — following the Block Library Extension (#69), I'd like to contribute a new
Computer Vision block and wanted to confirm the scope before packaging it.

### Block: Hand & Gesture Detector (MediaPipe Hand Landmarker)
VisualCircuit has FaceDetector (Haar) and ObjectDetector (YOLO) but no hand / landmark /
gesture block. This fills that gap and pairs naturally with the Controller blocks
(MotorDriver, Teleoperator) for gesture-driven robotics.

**What it does:** runs MediaPipe's Hand Landmarker on the input image to detect up to two
hands (21 landmarks each) and estimate a simple gesture (extended-finger count).

**Inputs:** BGR image (`Img`)
**Outputs (via `OutputType` parameter):**
- `image` — frame with hand landmarks, connections and a gesture label drawn on it
- `landmarks` — flat array of 21 (x, y) normalised landmarks + finger count, so downstream
blocks can react to gestures
**Parameter:** `OutputType` ('image' / 'landmarks')

**Implementation notes:**
- Uses MediaPipe's current **Tasks API** (`mediapipe.tasks.vision.HandLandmarker`) — the
legacy `mp.solutions.hands` API was removed in MediaPipe 0.10.30+, so the Tasks API is the
durable choice.
- Ships a bundled `hand_landmarker.task` model under `utils/models/`, mirroring how
FaceDetector/ObjectDetector bundle their Haar/YOLO models. One new dependency: `mediapipe`.
- Mirrors the existing block API (`inputs.read_image` / `outputs.share_image` /
`outputs.share_array` / `synchronise`).

I've prototyped and tested it (detects both hands and draws the 21-point skeleton on a real
frame). Once you're happy with the scope, I'll build it in the editor, export the `.vc3`, and
open a PR on VisualCircuit-resources with a short demo video per the custom-block guide.

Does this fit the library? Happy to adjust the I/O or naming.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the existing FaceDetector and ObjectDetector blocks, the custom-block guide, and the model location under utils/models/. Verify the MediaPipe Tasks API and the inputs.read_image, outputs.share_image, outputs.share_array, and synchronise entry points. Done means the new block supports both OutputType values, bundles hand_landmarker.task, and is exported as a .vc3 with the requested demo.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, robotics
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.