[vector_graphics] Luminance masks ignore the mask content's alpha: anti-aliased boundaries and content opacity are lost

## Summary

`vector_graphics` implements SVG luminance masking with a `ColorFilter.matrix` whose alpha row has a zero coefficient for the input alpha channel. Because `ColorFilter.matrix` evaluates on **unpremultiplied** color, the alpha of the mask content never contributes to the mask value.

Two consequences:

* **Anti-aliased mask boundaries are binarized.** For an opaque white mask shape — what Figma emits when a frame clips its contents — every edge pixel with nonzero raster coverage is promoted to full mask alpha regardless of coverage.
* **Explicit opacity on the mask content is ignored entirely.** A mask shape with `opacity="0.5"` produces a fully opaque result instead of a 50% one.

The same SVGs render correctly in browsers.

## Steps to reproduce

All three files use the identical path `M50 10L90 50L50 90L10 50Z` (a diamond, so the geometry is exactly reproducible outside the SVG pipeline too).

`mask.svg`:

```xml

```

`clip.svg` — visually equivalent, using `clipPath`:

```xml

```

`mask_opacity.svg` — same mask, with `opacity="0.5"` on the mask shape. Expected mask value is `luminance(white) × 0.5 = 0.5`:

```xml

```

## Measurements

Alpha along the horizontal scanline `y=30`, `x=26..36` (the diamond edge crosses at `x≈30`), plus the count of pixels with `0 < alpha < 255` over the whole 100×100 image. All rows come from a single run. The `dart:ui` rows drive `Canvas` directly with the same path, to isolate the mechanism from the SVG pipeline.

```
alpha at y=30, x=26..36 partial
flutter_svg mask.svg [0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255] 0
flutter_svg clip.svg [0, 0, 0, 128, 255, 255, 255, 255, 255, 255, 255] 160
flutter_svg mask_opacity.svg [0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255] 0
dart:ui current impl [0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255] 0
dart:ui fix A (two-pass dstIn) [0, 0, 0, 128, 255, 255, 255, 255, 255, 255, 255] 160
dart:ui fix B (black backdrop) [0, 0, 0, 128, 255, 255, 255, 255, 255, 255, 255] 160
dart:ui clipPath reference [0, 0, 0, 128, 255, 255, 255, 255, 255, 255, 255] 160
```

`mask.svg` contains **zero** anti-aliased pixels. `mask_opacity.svg` is pixel-identical to `mask.svg` — the interior reads `255` where `128` is expected, so the `opacity` attribute has no effect at all.

Left: `mask.svg`. Right: `clip.svg`. Top edge, 6× nearest-neighbor zoom.

## Root cause

[`packages/vector_graphics/lib/src/listener.dart`](https://github.com/flutter/packages/blob/main/packages/vector_graphics/lib/src/listener.dart) (v1.2.2):

```dart
static final Paint _grayscaleDstInPaint = Paint()
..blendMode = BlendMode.dstIn
..colorFilter = const ColorFilter.matrix([
0, 0, 0, 0, 0,
0, 0, 0, 0, 0,
0, 0, 0, 0, 0,
0.2126, 0.7152, 0.0722, 0, 0, // A' = 0.2126R + 0.7152G + 0.0722B + 0*A
]);

@override
void onMask() {
_canvas.saveLayer(null, _grayscaleDstInPaint);
}
```

[CSS Masking Module Level 1 §7.10.1 Mask processing](https://www.w3.org/TR/css-masking-1/#MaskValues) specifies this as two steps: compute luminance from non-premultiplied RGB using the `feColorMatrix` luminance-to-alpha coefficients, then multiply that luminance by the corresponding alpha value to obtain the mask value. The matrix above implements only the first step.

This would be harmless if the filter ran on premultiplied color: a 50%-covered white pixel is stored as `(0.5, 0.5, 0.5, 0.5)`, and `0.2126×0.5 + 0.7152×0.5 + 0.0722×0.5 = 0.5`. But the filter unpremultiplies first, so the same pixel is seen as `(1.0, 1.0, 1.0)` with `a = 0.5`, and the alpha is dropped:

```dart
const filter = ColorFilter.matrix([
0, 0, 0, 0, 0,
0, 0, 0, 0, 0,
0, 0, 0, 0, 0,
0.2126, 0.7152, 0.0722, 0, 0,
]);
// drawRect with Paint()..color = c ..colorFilter = filter, read back RGBA:
// 0xFFFFFFFF (opaque white) -> [0, 0, 0, 255]
// 0x80FFFFFF (white, a = 50%) -> [0, 0, 0, 255] // expected 128
// 0x40FFFFFF (white, a = 25%) -> [0, 0, 0, 255] // expected 64
// 0x00000000 (transparent) -> [0, 0, 0, 0]
```

An anti-aliased white path is pure white in unpremultiplied space at *any* nonzero coverage, so the result is a binary threshold at `coverage > 0`.

Note that a 4×5 affine color matrix cannot express `luminance × alpha` at all — it needs the products `R×A`, `G×A`, `B×A`. Changing the alpha row to `0.2126, 0.7152, 0.0722, 1, 0` yields `luminance + alpha`, not the product. So this is not fixable by adjusting coefficients.

## Why it is easy to miss

For an opaque white mask the interior stays correct and only the anti-aliased boundary visibly degrades, so the symptom looks like a generic "SVG renders jagged" problem rather than a masking bug. Explicit `opacity` is affected for the same reason, but that case is much more visible once you look for it.

Colored masks lose coverage identically; the resulting plateau simply sits at the shape's luminance rather than at 1.0. Gradients whose variation is encoded entirely in RGB luminance may appear mostly correct in their interior, but alpha-based variation and anti-aliased outer boundaries are still affected.

## Possible fixes

### A. Conceptually, apply the fully rendered mask twice — once for luminance, once for alpha

```dart
// in place of the single onMask() saveLayer
c.saveLayer(null, Paint()..blendMode = BlendMode.dstIn..colorFilter = _lumaToAlpha);
drawMask();
c.restore();
c.saveLayer(null, Paint()..blendMode = BlendMode.dstIn);
drawMask();
c.restore();
```

The first pass multiplies destination alpha by the luminance `L`, the second by the mask's alpha `A`, giving `dstAlpha × L × A`. This uses both the RGB and the alpha of the final mask image, so it generalizes to arbitrary mask content, and it needs no layer bounds.

The pseudocode is not a drop-in replacement: the current listener consumes mask drawing commands as a single forward stream, so this likely requires recording the mask content into an intermediate `Picture` and replaying it, or otherwise restructuring `onMask`.

### B. Fill the mask layer with opaque black before drawing the mask content

```dart
c.saveLayer(null, Paint()..blendMode = BlendMode.dstIn..colorFilter = _lumaToAlpha);
c.drawRect(layerBounds, Paint()..color = const Color(0xFF000000));
drawMask();
c.restore();
```

Compositing over opaque black turns premultiplied coverage into luminance: for a final mask color `(C, A)`, the result is `RGB = C×A, alpha = 1`, and `luminance(C×A) = luminance(C)×A`.

This is correct for ordinary source-over mask content and for the cases demonstrated here. However, making the backdrop opaque before the mask content is drawn changes what backdrop-sensitive operations inside the mask (blend modes, filters) see, so more complex mask content may need to be rendered into a transparent intermediate layer first. It also needs known layer bounds, whereas `saveLayer(null, ...)` is currently unbounded.

**Fix A is the safer general solution.**

## Workaround

If the mask is a single **opaque white** shape with no gradient or semi-transparency, replacing `` with `` (and `mask="url(#x)"` with `clip-path="url(#x)"`) is mathematically equivalent — `luminance × alpha = 1` inside, `0` outside — and restores anti-aliasing, since `onClipPath` uses `Canvas.clipPath` with `doAntiAlias: true`. This does not generalize to masks with gradients, semi-transparency, or non-white fills.

## Related

[#158734](https://github.com/flutter/flutter/issues/158734) `[flutter_svg] Support mask-type:alpha` — an adjacent problem arising from the same shared implementation, not the same bug. `mask-type` is not parsed anywhere in `vector_graphics_compiler`, `vector_graphics` or `flutter_svg` (verified against 1.2.6 / 1.2.2 / 2.3.0), so all masks go through this single luminance path. The refactoring needed here could also provide the basis for correct `mask-type: alpha` support, but that additionally requires parsing `mask-type` and selecting an alpha-only path that skips the luminance conversion.

## Version info

```
[!] Flutter (Channel stable, 3.38.5, on macOS 26.5.2 25F84 darwin-arm64, locale ja-JP) [542ms]
• Flutter version 3.38.5 on channel stable at
/Users/sho.ikeda/.local/share/mise/http-tarballs/13841a276204bcfbd65c66bf8a6cf984ee21a8886f73de06b44a188068889dcc
! Warning: `dart` on your path resolves to
/Users/sho.ikeda/.local/share/mise/http-tarballs/560ba7f51b0526f993ab3b6bcfc4348810ca2c3c5f6459aac85da97df4e6bed3/bin/dart, which is not inside your
current Flutter SDK checkout at /Users/sho.ikeda/.local/share/mise/http-tarballs/13841a276204bcfbd65c66bf8a6cf984ee21a8886f73de06b44a188068889dcc.
Consider adding /Users/sho.ikeda/.local/share/mise/http-tarballs/13841a276204bcfbd65c66bf8a6cf984ee21a8886f73de06b44a188068889dcc/bin to the front of
your path.
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision f6ff1529fd (8 months ago), 2025-12-11 11:50:07 -0500
• Engine revision 1527ae0ec5
• Dart version 3.10.4
• DevTools version 2.51.1
• Feature flags: enable-web, enable-linux-desktop, enable-macos-desktop, enable-windows-desktop, enable-android, enable-ios, cli-animations,
enable-native-assets, omit-legacy-version-file, enable-lldb-debugging
• If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✗] Android toolchain - develop for Android devices [406ms]
✗ ANDROID_HOME = /Users/sho.ikeda/.local/share/mise/installs/android-sdk/1.0
but Android SDK not found at this location.

[✓] Xcode - develop for iOS and macOS (Xcode 26.5) [6.4s]
• Xcode at /Applications/Xcode-26.5.0.app/Contents/Developer
• Build 17F42
• CocoaPods version 1.16.2

[✓] Chrome - develop for the web [9ms]
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Connected device (3 available) [14.0s]
• iPhone 17 (mobile) • 197E0FAC-39AA-4E1A-83AA-1028496BCABB • ios • com.apple.CoreSimulator.SimRuntime.iOS-26-5 (simulator)
• macOS (desktop) • macos • darwin-arm64 • macOS 26.5.2 25F84 darwin-arm64
• Chrome (web) • chrome • web-javascript • Google Chrome 150.0.7871.187

[✓] Network resources [799ms]
• All expected network resources are available.
```

This appears to be an algorithmic issue in the mask implementation rather than an anti-aliasing configuration issue. It reproduces with the Skia software renderer via `flutter_test`, and the same failure follows on any backend that evaluates the color matrix in unpremultiplied color space.

Start in packages/vector_graphics/lib/src/listener.dart at _grayscaleDstInPaint and onMask(), then compare the current ColorFilter.matrix behavior with the CSS Masking processing described here. Evaluate the proposed two-pass and black-backdrop approaches against the listed dart:ui measurements. Done means mask alpha, anti-aliased boundaries, and explicit opacity are preserved without regressing the clipPath reference behavior.

Written by the indexing model from the issue text.