BabylonJS / BabylonJS/BabylonNative

[Android][OpenGL] Support GL_TEXTURE_EXTERNAL_OES in ExternalTexture

Open
#1,847 0 comments 0 reactions 1 assignee Claimed by @bkaradzic-microsoft View on GitHub
Dominant language
C++
Stars
919
Forks
162
Avg merge
1d 15h
Merged PRs (30d)
19

Description

**For questions, please use the [forum](https://forum.babylonjs.com)**.

**Is your feature request related to a problem? Please describe.**

Android commonly delivers camera and hardware-decoded video frames through
`SurfaceTexture`. The underlying OpenGL texture uses the
`GL_TEXTURE_EXTERNAL_OES` target rather than `GL_TEXTURE_2D`.

These textures cannot be sampled as ordinary 2D textures:

- They must be bound using `GL_TEXTURE_EXTERNAL_OES`.
- GLES 3 shaders must enable `GL_OES_EGL_image_external_essl3`.
- They must use `samplerExternalOES`.
- They are sampled-only, single-level textures with restricted sampler state.

BabylonNative's current OpenGL `ExternalTexture` path cannot describe this
texture kind:

- [`Graphics::GL::Texture::Descriptor`](https://github.com/BabylonJS/BabylonNative/blob/master/Core/Graphics/Include/RendererType/OpenGL/Babylon/Graphics/GL/Texture.h)
has no texture-target/type field. Its target is currently derived from the
layer count as either a 2D texture or a 2D-array texture.
- [`ExternalTexture`](https://github.com/BabylonJS/BabylonNative/blob/master/Plugins/ExternalTexture/Source/ExternalTexture_Shared.h)
always creates a bgfx texture through `bgfx::createTexture2D`.

As a result, an externally owned OES texture cannot be wrapped and sampled
directly by BabylonNative. The application must currently perform an additional
full-frame GPU pass:

```text
SurfaceTexture / MediaCodec / camera
-> GL_TEXTURE_EXTERNAL_OES
-> samplerExternalOES full-screen draw
-> RGBA GL_TEXTURE_2D
-> BabylonNative ExternalTexture
```

BabylonNative's own Android
[`NativeCamera`](https://github.com/BabylonJS/BabylonNative/blob/master/Plugins/NativeCamera/Source/Android/CameraDevice.cpp)
currently demonstrates this workaround: it samples the OES camera texture into
an RGBA 2D framebuffer texture before making that texture available to bgfx.

Although this avoids CPU readback, it still requires an extra destination
texture and a full-frame render pass for every video frame, adding GPU bandwidth,
synchronization, and latency.

**Describe the solution you'd like**

Allow the OpenGL `ExternalTexture` integration to represent and sample an
externally owned `GL_TEXTURE_EXTERNAL_OES` texture directly.

The exact API shape is open for discussion. One possible approach would be to
extend `Graphics::GL::Texture::Descriptor`, or an equivalent external-texture
descriptor, with a texture kind/target such as `ExternalOES`.

The resulting texture should:

- Be bindable as `GL_TEXTURE_EXTERNAL_OES`.
- Be sampled through `samplerExternalOES` on OpenGL ES.
- Be usable from JavaScript after `ExternalTexture::CreateForJavaScript` and
`engine.wrapNativeTexture`, without first copying it into a
`GL_TEXTURE_2D`.
- Support replacing the underlying OES handle through
`ExternalTexture::Update`, if handle replacement is otherwise supported.
- Preserve the existing externally owned lifetime model; BabylonNative must
not delete a texture owned by `SurfaceTexture` or another producer.
- Detect unavailable OES/ESSL3 extension support and report a clear error.

The implementation should also enforce or document the restrictions of an
external OES texture:

- Sampled-only; not a Babylon render target.
- One layer and base mip level zero.
- No mipmap generation or mipmapped filtering.
- `CLAMP_TO_EDGE` wrapping.
- `NEAREST` or `LINEAR` minification/magnification filtering.

The producer would remain responsible for its platform lifecycle, including
calling `SurfaceTexture.updateTexImage()` on the correct GL context and applying
the matrix returned by `SurfaceTexture.getTransformMatrix()`.

Suggested acceptance criteria:

1. An Android `SurfaceTexture`-backed OES texture containing a known test image
can be wrapped as an `ExternalTexture`.
2. A Babylon material can sample and display it without an intermediate
OES-to-RGBA texture copy.
3. Updating the `SurfaceTexture` updates the displayed Babylon texture.
4. Externally owned texture lifetime and graphics-thread requirements remain
correct.
5. Existing 2D, 2D-array, D3D, and Metal external-texture paths are unchanged.
6. Unsupported sampler states or missing OES extensions fail clearly rather
than rendering a black texture.

**Describe alternatives you've considered**

1. **Render OES into an RGBA `GL_TEXTURE_2D`.**
This is the current working solution, but it adds one full-frame texture and
GPU render pass per input frame.

2. **Read the decoded frame back to the CPU and upload it again.**
This is substantially more expensive and defeats the purpose of decoding to
a `Surface`.

3. **Render the OES texture outside BabylonNative using custom GLES code.**
This avoids the copy but prevents the texture from participating normally
in Babylon materials, scene composition, and texture lifecycle management.

Requesting a regular 2D texture from the producer is generally not an option
for Android `SurfaceTexture`/`MediaCodec` output, whose texture target is
defined as `GL_TEXTURE_EXTERNAL_OES`.

**Additional context**

- Android `SurfaceTexture` documentation:
https://developer.android.com/reference/android/graphics/SurfaceTexture
- Khronos `GL_OES_EGL_image_external`:
https://registry.khronos.org/OpenGL/extensions/OES/OES_EGL_image_external.txt
- Khronos GLES 3 shader support:
https://registry.khronos.org/OpenGL/extensions/OES/OES_EGL_image_external_essl3.txt
- The OpenGL `ExternalTexture` backend was introduced in:
https://github.com/BabylonJS/BabylonNative/pull/1780
- bgfx has historical design context for this capability:
- https://github.com/bkaradzic/bgfx/issues/1069
- https://github.com/bkaradzic/bgfx/pull/2021

The historical bgfx work used a target-aware `overrideInternal` operation and an
external-sampler shader abstraction. The currently exposed bgfx API used by
BabylonNative does not appear to retain that target parameter, so implementing
this may require coordinated bgfx and BabylonNative changes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.