microsoft / microsoft/microsoft-ui-xaml

Lifted Microsoft.UI.Composition still rejects custom/D2D effects while MUC also lacks Visual-to-GPU interop

Open
#11,813 1 comment 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
C++
Stars
8.4k
Forks
942
Avg merge
2d 7h
Merged PRs (30d)
105

Description

### Describe the bug

WinUI 3 lifted Composition has no public path to apply custom GPU effects to Composition content (UWP capability regression)

WinUI 3 currently provides no supported way to apply an application-defined GPU effect to arbitrary "Microsoft.UI.Composition" content.

There are two independent limitations which together create a capability regression compared with UWP / "Windows.UI.Composition":

1. "Microsoft.UI.Composition.Compositor.CreateEffectFactory()" still only accepts a hard-coded subset of Direct2D effects. Effects documented by Win2D as "[NoComposition]", as well as application-defined Direct2D effects and "PixelShaderEffect", are rejected even when the application correctly implements "IGraphicsEffect", "IGraphicsEffectSource", and "IGraphicsEffectD2D1Interop".

2. Unlike "Windows.UI.Composition.Visual", a "Microsoft.UI.Composition.Visual" cannot be passed to "GraphicsCaptureItem.CreateFromVisual()", so WinUI 3 also lacks the UWP escape path of exporting a Composition visual into an application-owned D3D texture and processing it externally.

### Why is this important?

This means WinUI 3 currently has neither of these supported paths:
```
Composition source

custom shader inside Composition

nor:

Microsoft.UI.Composition.Visual

application-readable GPU surface

custom shader outside Composition
```
This is especially surprising now that WinUI 3 uses a lifted Composition implementation with an in-process composition engine. It no longer considers privacy implications when composing UWP content with the system DWM handle.

### Steps to reproduce the bug

A developer can implement a minimal graphics effect wrapper without using Win2D:
```cpp
struct Effect :
winrt::implements<
Effect,
IGraphicsEffect,
IGraphicsEffectSource,
IGraphicsEffectD2D1Interop>
{
HRESULT GetEffectId(GUID* id) noexcept override
{
*id = CLSID_D2D1GaussianBlur;
return S_OK;
}

// Implement properties/sources...
};
```
Using a Composition-supported CLSID such as CLSID_D2D1GaussianBlur allows the graph to be created successfully.

Using the same implementation but changing the effect ID to a valid Direct2D effect which Win2D marks "[NoComposition]", for example:

CLSID_D2D1DisplacementMap

causes the Composition effect creation path to fail.

The same occurs for application-defined Direct2D effects registered with "ID2D1Factory1" and for Win2D "PixelShaderEffect".

This demonstrates that the failure is not caused by the Win2D effect wrapper. The "IGraphicsEffectD2D1Interop" object is only describing the effect correctly; the lifted Composition implementation is rejecting the effect ID because there is no public registration/extensibility mechanism corresponding to Direct2D's custom-effect registration support.

### Actual behavior

"Microsoft.UI.Composition" only accepts its internal fixed set of effect IDs.

Application-defined Direct2D effects and "[NoComposition]" effects fail when the Composition effect factory/brush is created.

At the same time, WinUI 3 provides no equivalent to `GraphicsCaptureItem::CreateFromVisual(Windows::UI::Composition::Visual);` for `Microsoft::UI::Composition::Visual` so developers cannot move the visual content to an application-owned D3D pipeline either.

The practical result is that implementing effects such as:

- spatially varying refraction;
- displacement mapping;
- arbitrary convolution;
- custom blur algorithms;
- chromatic dispersion;
- custom material shaders;

over normal WinUI 3 Composition/XAML content requires either:

- an additional whole-window/capture pipeline with different semantics and latency, where possible; or
- reverse engineering and hooking private lifted Composition effect/compiler internals.

A proof-of-concept implementation of the latter approach demonstrates that custom HLSL can in fact execute as a normal Composition effect node and consume a "CompositionBackdropBrush", but it requires private ABI hooks which break across Windows App SDK runtime versions.

### Expected behavior

The lifted WinUI 3 Composition implementation should provide at least one supported path for this scenario.

Preferably, the in-process Composition engine should expose a supported custom-effect/pixel-shader API, for example an API equivalent in capability to Win2D "PixelShaderEffect", allowing application shader bytecode, source mappings, sampler metadata, constants, and bounds information to participate directly in a Composition effect graph.

For example:
```
CompositionBackdropBrush

Application-defined pixel shader

CompositionEffectBrush
```
without an additional capture round trip.

If arbitrary shader execution cannot be supported by a particular composition engine, this could be exposed as an engine capability. For example, custom shaders could initially be supported only by the lifted in-process engine while remaining unsupported for privileged/system-engine execution.

Separately, WinUI 3 should restore the Visual-to-GPU interoperability capability available to WUC, either through: `GraphicsCaptureItem.CreateFromVisual(Microsoft.UI.Composition.Visual)` or an equivalent Windows App SDK-owned API that exposes a "Microsoft.UI.Composition.Visual" subt

### Screenshots

_No response_

### NuGet package version

2.4.0

### Windows version

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the minimal C++ IGraphicsEffect/IGraphicsEffectSource/IGraphicsEffectD2D1Interop reproduction and the documented Composition effect-factory and GraphicsCaptureItem APIs described in the issue. Compare supported and rejected effect IDs, including CLSID_D2D1DisplacementMap, and investigate the Visual-to-GPU interop gap. Done requires a supported path for custom Composition effects or equivalent visual export, with the capability and API scope resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
computer-graphics, desktop
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.