webarkit / webarkit/jsfeatNext

examples: end-to-end AR demo anchoring a 3D object via pose_estimator

Open
#179 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
12
Forks
4
Avg merge
16h 24m
Merged PRs (30d)
34

Description

Summary

Now that pose_estimator exists (#83), an example should show the full natural-feature AR pipeline end to end: detect → describe → match (#133 bfmatcher) → homography (motion_estimator) → camera pose (R, t) → a 3D object anchored on the target.

Today the ORB samples (sample_orb.html, sample_orb_pinball.html) stop at the homography: find_transform fits H and render_pattern_shape warps the target outline into a flat 2D quad. pose_estimator provides the missing H → [R | t] step, but nothing demonstrates it.

This is the visual counterpart to #83, deliberately split out (same way #176 followed #133).

Why this is its own issue, not part of #83

Rendering a 3D pose needs renderer glue that jsfeatNext deliberately does not ship. pose_estimator.estimate stops at (R, t) in the OpenCV camera frame. Putting a 3D object on screen needs:

  • modelViewGL(pose) — a column-major 4×4 modelview for WebGL/three.js, with the CV→GL axis flip (diag(1, -1, -1)).
  • projectionGL(K, near, far) — a GL projection from the intrinsics.

#83 moved these up to the high-level AR layer's renderer adapters (#97) on purpose, to keep jsfeatNext renderer-agnostic. The reference implementations are kept in #83's prototype <details> block for whoever writes this.

So the example itself carries that glue (it's an example, not library code) — which means this should land after #96/#97 settle where the adapter boundary is, so the example's glue matches the eventual adapter rather than being written now and rewritten later.

Prerequisite (same as #176)

The examples load ../dist/jsfeatNext.mjs, and dist/ is only rebuilt at release time (per MAINTAINERS.md; #143 tracks dropping it from version control). So pose_estimator reaches the example after a release that includes #83, or alongside a deliberate local dist/ rebuild for verification.

Sketch

Two shapes to weigh at implementation:

  1. A new examples/sample_pose.html — cleanest: a dedicated demo that tracks a planar target and anchors a simple 3D object (a cube/axes) on it via three.js or raw WebGL. Doesn't disturb the existing ORB samples.
  2. Extend sample_orb_pinball.html — reuse its existing tracking, add the pose + render step. Less new code, but conflates two demos.

Either way the pose math is estimator.estimate(H, pose); the new work is the renderer glue and a minimal 3D scene, plus pose_estimator.intrinsics(w, h) for the uncalibrated bootstrap.

Acceptance criteria

  • An example anchors a 3D object on a tracked planar target using jsfeat.pose_estimator.
  • The CV→GL renderer glue (modelViewGL/projectionGL) lives in the example, not in src/ — jsfeatNext stays renderer-agnostic.
  • Uses pose_estimator.intrinsics(w, h) for the bootstrap K, or documents where a real calibration would plug in.
  • Verified live in a browser with a webcam (like #177).

Related

  • Provides the pose primitive: #83
  • Sibling example follow-up (bfmatcher): #176 / #177
  • Where the renderer-adapter boundary is decided: #96, #97
  • dist/ prerequisite: #143
  • Plan: docs/roadmap-1.0.0.md

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

First review #96/#97 and the prototype details in #83 to confirm the renderer-adapter boundary, then inspect sample_orb.html and sample_orb_pinball.html for the existing tracking flow. Verify that the released or locally rebuilt dist/jsfeatNext.mjs exposes pose_estimator. Done means a browser webcam demo anchors a 3D object to a planar target, keeps modelViewGL/projectionGL in the example, and uses or documents intrinsics(w, h).

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ar-vr-xr, computer-vision, web-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.