Flutter GPU: external render targets for OpenXR — prototype and request for upstream guidance

Hi Flutter team, I would like your guidance on contributing a Flutter GPU/engine change that lets Flutter render into GPU images owned by another system. I have a working Meta Quest 3 prototype in the forks lsited bellow. Sorry for the sizeable patch and long post!

**TL;DR: I added a generic external render-target API to a Flutter engine fork, allowing Flutter Scene to render directly into OpenXR’s left- and right-eye images.** This removes the intermediate Flutter Canvas presentation and native eye copy, while keeping Flutter UI panels separate. My simplified demo went from roughly **52–54 FPS to around 70 FPS**, although this was not a controlled Canvas-only benchmark. **I’m asking both the Flutter and Flutter Scene maintainers how to revise these forks into suitable upstream contributions.**

- **Flutter engine changes:** [Generic external GPU render surfaces](https://github.com/flutter/flutter/compare/main...adrian-moisa:flutter_vr:main) — 25 files · +2.4K / −6 lines
- **Flutter Scene integration:** [OpenXR support and Quest example gallery](https://github.com/bdero/flutter_scene/compare/master...adrian-moisa:flutter_scene_vr:add-openxr-support)
- **Background and discussion:** [Flutter Scene issue #365](https://github.com/bdero/flutter_scene/issues/365)

## Why I started this

I’m developing [Visual Space](https://visualspace.app), a Flutter application with a spatial UI and a 3D editor. We use Flutter Scene for ordinary 3D rendering, but initially built a separate native renderer for immersive Quest support, with Flutter supplying the interactive UI panels.

Before this work, **we had already built our own custom native 3D renderer for Quest**, with Flutter supplying the UI panels. It works, but leaves us maintaining two rendering engines: Flutter Scene for web and the ordinary application, and our own native renderer for VR. Every material, lighting effect, shadow improvement, and new scene feature then needs work in both places. **We want to replace that duplicated native renderer with Flutter Scene**, while keeping OpenXR responsible for tracking and headset presentation.

Getting Flutter Scene running in VR was the next step. Our first prototype rendered its left- and right-eye views through Flutter Canvas into a large Android surface, then copied those images into OpenXR’s eye textures. This let us reuse Flutter Scene, but added composition, buffering, and copying work to every stereo frame.

To remove that extra path, **Flutter Scene needs permission to draw directly into the GPU eye images that OpenXR owns**. That is the missing capability addressed by the Flutter engine patch. I added an API that lets Flutter GPU temporarily borrow those images as rendering destinations, then built a custom local Flutter engine to try it. This lets us keep Flutter Scene as the renderer without routing the stereo world through Canvas.

## What the Flutter patch adds

The engine contribution is **generic external GPU output**, not Quest-specific session or controller code. OpenXR is the first consumer I have working.

The main pieces are:

- **A new destination for Flutter GPU rendering.** `GpuExternalSurface` and its frame type use the existing `GpuSurface` / `GpuSurfaceFrame` pattern. Native code supplies a target; Dart acquires it, draws into it, and presents or discards the frame. Flutter borrows the target rather than taking ownership of its GPU storage.
- **Native registration and frame delivery.** Embedder C APIs register surfaces, supply frames, and unregister them. An Android bridge through `FlutterRenderer` and `FlutterJNI` exposes the same capability to an Android host.
- **Shared GLES resources.** The Android bridge allows the host and Flutter to access the same GPU textures. It creates the framebuffer used for drawing inside Flutter’s own graphics context, because framebuffer objects cannot simply be shared between contexts the way textures can.
- **Explicit completion and cleanup.** Completion is tied to the commands that write the final image. Sending commands alone does not mean OpenXR can safely reuse it. Android uses GPU fences to detect when drawing has finished where supported, with a conservative blocking fallback.

The borrowed target is currently a single-sample final color destination. Flutter Scene still owns intermediate buffers for depth, shadows, antialiasing, HDR, and effects.

In the Flutter Scene fork, `Scene.renderViewsToTargets` connects the renderer to these targets. The OpenXR companion package handles the headset session, tracking, controllers, and eye-image coordination, outside the engine patch.

## Who drives the frames?

The native OpenXR host waits for the headset’s next frame using `xrWaitFrame`, acquires both eye images, and requests stereo rendering through the Android/Dart bridge. Flutter Scene updates the scene once and renders both eyes. The host waits for both GPU-completion callbacks before returning the images to OpenXR for display.

**The stereo world does not wait for a Flutter widget paint.** UI panels update separately. This uses a custom Android activity/plugin alongside Flutter’s existing Android embedding; I have not replaced Flutter’s overall frame scheduler.

Is this separation appropriate, or would a dedicated OpenXR embedder with broader control over Flutter’s scheduling be a better approach?

## What works, and what still needs work

I have built and tried this on **Meta Quest 3 using Android/OpenGL ES**. The gallery includes real Flutter Scene examples, interactive panels, and performance controls. Vulkan and Metal are not implemented for this path, and each eye is rendered separately.

The FPS improvement above followed earlier shadow tuning, and the direct path changed scheduling as well as presentation. Heavy scenes and effects still reduce performance. A fast OpenXR submission loop also does not necessarily mean equally fast fresh scene content.

Visual Space sometimes has **up to three layers of Flutter panels**. We already suspend texture updates for panels in “sleep mode” to preserve performance. Separating stereo rendering leaves more room and budget for that UI atlas, although it does not solve independent panel updates by itself.

**The engine patch is not ready to merge unchanged.** An initial review flagged concerns around access from the correct engine threads, cleanup and shared resources when multiple Flutter engines are involved, and handling the extra GPU attachments used for depth and antialiasing. These need investigation, fixes, and focused engine tests. The current engine contribution does not yet include those tests; successful builds and a working headset demo are not a substitute for them.

The [Flutter fork README](https://github.com/adrian-moisa/flutter_vr/blob/main/README.md) and [Flutter Scene setup guide](https://github.com/adrian-moisa/flutter_scene_vr/blob/add-openxr-support/packages/flutter_scene/README.md) cover building the custom engine and running the demo. Profiling additions and fork documentation could be separated from the core API patch.

## Guidance I’m asking for

**We are waiting for guidance from both repositories so we can adjust the engine API and Flutter Scene integration together**, then submit focused contributions for review. In particular:

1. Does a generic external destination extending `GpuSurface` look like an acceptable direction for Flutter GPU?
2. Where should the Android resource-sharing and frame-completion integration live, and is the current host approach appropriate?
3. Would you consider an initial GLES implementation, with other backends explicitly unsupported, or should the first proposal cover a wider backend contract?
4. What lifecycle guarantees, tests, and API changes would you need before reviewing this for inclusion? Who would be the right maintainers to involve?

I understand this capability was not planned, and I’m happy to split the patch into smaller changes. I have substantial programming experience, but I’m not a graphics-engine specialist. Help with the design, review, testing, and contribution process would be very welcome; I’m currently handling this work on my own alongside the application.

Both forks are already part of our development stack, and we are committed to this work. Next comes integrating the direct renderer into Visual Space itself. Until these changes can land upstream, we will maintain and rebase the forks, but I hope that does not have to become a permanent second project.

Thank you for reading, and for any guidance or help getting this into a form the Flutter and Flutter Scene teams could accept.

Start with the Flutter fork README and the named GpuExternalSurface, FlutterRenderer, FlutterJNI, and Scene.renderViewsToTargets entry points, then run the existing Meta Quest 3 GLES demo. Review the concerns about engine threads, shared-resource cleanup, GPU attachments, and missing focused tests. Done means maintainers provide a direction for a smaller, reviewable upstream contribution.

Written by the indexing model from the issue text.