[Bug] Issues with Masking Extension in deck.gl for Vector Tiles

Description

The Masking Extension in deck.gl does not behave as expected when used with MVTLayer. Specifically, when attempting to use the Mask Extension to control the visibility of certain areas based on another layer, both layers become completely hidden regardless of the specified settings.

Flavors

• Script tag
• React
• Python/Jupyter notebook
• MapboxOverlay
• GoogleMapsOverlay
• CartoLayer
• ArcGIS

Expected Behavior

The expected behavior is that the areasLayer should act as a mask for the buildingsLayer, allowing only the areas covered by areasLayer to display buildingsLayer data. The rest of the buildingsLayer should be masked out.

Steps to Reproduce

1. Set up a basic Mapbox instance with deck.gl overlay.
2. Define two MVTLayers: one for the mask (areasLayer) and one to be masked (buildingsLayer).
3. Apply the Mask Extension to areasLayer with appropriate maskId.
4. Observe that instead of masking, both layers are completely hidden.

Pen: https://codepen.io/JacobWeinbren/pen/yLWJWVP

<div id="map" class="w-full h-full"></div>

<script>
	import { Deck } from "@deck.gl/core";
	import { MapboxOverlay } from '@deck.gl/mapbox';
	import { MVTLayer } from "@deck.gl/geo-layers";
	import { MaskExtension } from '@deck.gl/extensions';
	import mapboxgl from 'mapbox-gl';
	import { GL } from "@luma.gl/constants";

	const MAPBOX_TOKEN = import.meta.env.PUBLIC_MAPBOX_TOKEN;

	// Initialize Mapbox map
	const map = new mapboxgl.Map({
		container: 'map',
		style: "mapbox://styles/mapbox/dark-v11",
		center: [-4.2026, 56.4907],
		zoom: 6,
		accessToken: MAPBOX_TOKEN,
	});

	map.on('load', () => {
		const firstLabelLayerId = map
				.getStyle()
				.layers.find((layer) => layer.type === "symbol").id;

		console.log(firstLabelLayerId);

		// Define the mask layer
		const areasLayer = new MVTLayer({
			id: "mask",
			data: "https://map.jacobweinbren.workers.dev/scottish-areas-ages/{z}/{x}/{y}.mvt",
			minZoom: 0,
			maxZoom: 22,
			getFillColor: [255, 255, 255, 255],
			beforeId: firstLabelLayerId,
			extensions: [new MaskExtension({ maskId: 'mask-layer' })],
			maskId: 'mask-layer',
		});

		// Define the intersected layer
		const buildingsLayer = new MVTLayer({
			id: "mask-layer",
			data: "https://map.jacobweinbren.workers.dev/scottish-intersected-ages/{z}/{x}/{y}.mvt",
			minZoom: 0,
			maxZoom: 22,
			operation: 'mask',
			beforeId: firstLabelLayerId
		});

		const deckOverlay = new MapboxOverlay({
			interleaved: true,
			layers: [areasLayer, buildingsLayer],
		});

		map.addControl(deckOverlay);
	});
</script>

Environment

Framework version: deck.gl@9.0.14
Browser: Chrome 110.0

Logs

No response

Start by running the linked CodePen with deck.gl@9.0.14 and compare the areasLayer and buildingsLayer behavior. Inspect the MaskExtension, MVTLayer, and MapboxOverlay entry points involved in the reproduction. Done means the areasLayer masks only the intended buildings while both layers remain visible outside the masked region.