webarkit / webarkit/webarkit-testing

3D pose renders behind the camera (CV vs GL convention); matrixGL_RH throws

Open
#31 1 comment 0 reactions 1 assignee View on GitHub

@kalwalt is already working on this.

Since May 29, 2026.

bug Emscripten / C / C++ enhancement javascript
Dominant language
JavaScript
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Summary

When a marker is successfully tracked, the getMarker event fires with a valid pose (real rotation + translation), but three.js examples still show no 3D object. The object is placed behind the camera and gets frustum-clipped.

Surfaced while implementing #30 (static-image Teblid example): the marker tracks correctly (Marker tracked ! Num. matches : 1000) and a found pose arrives, yet no sphere is visible.

Evidence (runtime)

Sample pose delivered to the scene (three.js column-major elements):

translation = (0.953, -0.384, +5.824)

Projection matrix from the same run (column-major):

[-1.785, 0, 0, 0,  0, 2.380, 0, 0,  0, 0, -1.0002, -1,  0, 0, -0.20002, 0]

This is a standard OpenGL projection (proj[11] = -1, camera looks down −Z). Plugging the object origin (0.953, -0.384, 5.824, 1) through it:

  • clip.w = -view.z = -5.824 (negative → behind camera)
  • ndc.z ≈ 1.03 (> 1 → beyond far plane)

So the object is clipped and never rasterized, despite correct tracking.

Root cause

The pose field (from getPoseMatrix()transMatToGLMat()) is in OpenCV camera convention (object in front ⇒ +Z), which is incompatible with the OpenGL projection used for rendering (object in front ⇒ −Z). The right-handed GL matrix matrixGL_RH (arglCameraViewRHf) — the one that should be fed to three.js — currently throws an exception (see recent dev commits experimenting with the pose matrix).

Suggested direction

  • Provide a correct GL modelview/pose matrix (apply the full CV→GL handedness flip: negate Y and Z appropriately on both rotation and translation), or fix arglCameraViewRHf / matrixGL_RH so it no longer throws and can be used directly.
  • Quick verification: negating the translation Z on the pose before setMatrix(root.matrix, ...) should bring the object into view — useful as a confirmation while the proper conversion is sorted out.

Notes

  • The fix likely lives in the emscripten/WebARKitLib submodule (pose conversion: invertPose/cvToGl/arglCameraViewRHf) and would require rebuilding dist/WebARKit.js.
  • Affects all three.js worker examples, not just the static-image one from #30.
  • Tracking itself is working; this is purely a pose-convention / rendering issue.

Related latent bug (already worked around in #30)

WebARKitTracker::processFrame calls GetTrackedFeaturesWarped() when a marker is detected on the very first frame (_frameCount == 0), before GetInitialFeatures() has populated the tracked-feature set. perspectiveTransform then runs on an empty point set and throws (CV_Assert(scn + 1 == m.cols)). The #30 static example works around this by feeding one blank warmup frame, but the library should guard the pose block against an empty feature set.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.