BabylonJS / BabylonJS/BabylonNative

iOS/JSC: getUserMedia camera VideoTexture never renders — NativeVideo::SetSrcObject rejects MediaStream (InstanceOf fails across napi/JSC)

Open
#1,768 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
919
Forks
162
Avg merge
1d 15h
Merged PRs (30d)
19

Description

### Summary

On iOS (JavaScriptCore), `NativeVideo::SetSrcObject` rejects the `MediaStream` returned by `navigator.mediaDevices.getUserMedia`, so `VideoTexture.CreateFromStreamAsync` never renders the camera feed. Camera *capture* works (AVCaptureSession runs, frames flow), but no texture ever updates.

### Environment
- Babylon Native `master` @ `64f545de`; Babylon.js `9.9.1` (UMD `babylon.max.js`)
- iOS 26.5, iPhone 15 Pro Max (device), Xcode 26
- Repro: `Apps/Playground` iOS with the `cameraTexture` path in `experience.js` enabled

### Root cause

`Plugins/NativeCamera/Source/NativeVideo.cpp`, `SetSrcObject`:

```cpp
if (value.IsNull() || value.IsUndefined() ||
!value.As().InstanceOf(MediaStream::GetConstructor(env)))
{
// reject: m_streamObject cleared, m_isReady = false
return;
}
```

`getUserMedia` creates the stream via `MediaStream::GetConstructor(env).New({})`, and `SetSrcObject` checks `InstanceOf(MediaStream::GetConstructor(env))` against the same cached constructor — yet **`InstanceOf` returns false** across the napi/JavaScriptCore boundary for the genuine `MediaStream` object.

On-device trace (instrumented):
```
SetSrcObject called null=0 obj=1 instanceOfMediaStream=0 <- object is a MediaStream, InstanceOf fails
```

Because the stream is rejected, `readyState` stays `0`, so Babylon.js's `VideoTexture.CreateFromStreamAsync` never gets the `"playing"` event (its `NativeVideo::Play` only fires `"playing"` when a stream is set), the returned Promise never resolves, the `VideoTexture` is never constructed, and `engine.updateVideoTexture` is never called.

### Impact
`getUserMedia` + `VideoTexture.CreateFromStreamAsync` (camera-to-texture) is completely broken on the JSC/iOS backend.

### Suggested fix
Either investigate why `Napi::Value::InstanceOf` fails for ObjectWrap constructors on the JSC engine, or identify the `MediaStream` more robustly. A pragmatic fix is to duck-type on the presence of `getVideoTracks`:

```cpp
const bool looksLikeMediaStream =
value.IsObject() && value.As().Get("getVideoTracks").IsFunction();
if (value.IsNull() || value.IsUndefined() || !looksLikeMediaStream) { /* reject */ }
```

Happy to open a PR.

Contributor guide

Open the contributing guide

Research direction

Start with Apps/Playground on iOS using the cameraTexture path in experience.js, then inspect NativeVideo::SetSrcObject in Plugins/NativeCamera/Source/NativeVideo.cpp and trace the failing InstanceOf check on JavaScriptCore. Done means the genuine getUserMedia MediaStream is accepted, the playing event occurs, CreateFromStreamAsync resolves, and the camera VideoTexture renders on iOS.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, ios, javascript
Domain
computer-graphics, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.