BabylonJS / BabylonJS/BabylonNative

Canvas: clip() does not clip to the path (needs real stencil-based path clipping)

Open
#1,845 0 comments 0 reactions 1 assignee Claimed by @bkaradzic-microsoft View on GitHub
Dominant language
C++
Stars
919
Forks
162
Avg merge
1d 15h
Merged PRs (30d)
19

Description

Follow-up to #1824 (see the `clip()` discussion there) and split out of the follow-up work in #1843 / #1844.

## Problem

`CanvasRenderingContext2D.clip()` does not clip to the path. `Context::Clip` has two branches, and neither is correct:

- **Rectangular path** -- intersects an `nvgScissor` with the path's bounding box. Correct only when the path really is an axis-aligned rectangle.
- **Non-rectangular path** (`m_pathHasNonRect`) -- leaves the path current and lets the next fill draw it, then leaves the enclosing scissor untouched. This does not clip at all; it just draws the clip path.

`roundRect()` is a third case: it deliberately does **not** set `m_pathHasNonRect`, so it takes the scissor branch and silently loses its rounded corners. That is deliberate, and the comment in `Context::RoundRect` explains why -- routing it into the emulation is much worse, because nanovg fills the **union** of subpaths rather than their intersection:

```js
ctx.roundRect(x, y, w, h, 20);
ctx.clip();
ctx.fillRect(0, 0, 1000, 1000); // paints the whole fillRect, not the rounded region
```

Measured on the "Native Canvas" visual test: the scissor's square bounding box gives **1.850%** pixel difference, routing `roundRect` into the union emulation gives **20.980%**. The bounding box is the smaller error because it can only ever clip *too little*, whereas the union failure is unbounded.

`save()`/`restore()` also do not currently rewind a clip region, which the spec requires.

## Why it is not a small fix

The good news is that no framebuffer work is needed: both the canvas framebuffer (`Canvas.cpp`) and the filter-stack pool (`FrameBufferPool.cpp`) already allocate `D24S8`, so a stencil buffer is present everywhere nanovg draws.

The obstacle is that **bgfx's stencil state has no write mask** -- `setStencil` offers `TEST_*`, `FUNC_REF`, `FUNC_RMASK` and the three op slots, but no equivalent of `glStencilMask`. That rules out the usual "reserve a high bit for the clip" scheme, because the existing concave-fill cover pass in `glnvg__fill` zeroes the entire stencil value:

```cpp
gl->encoder->setStencil(0
| BGFX_STENCIL_TEST_NOTEQUAL
| BGFX_STENCIL_FUNC_RMASK(0xff)
| BGFX_STENCIL_OP_FAIL_S_ZERO
| BGFX_STENCIL_OP_FAIL_Z_ZERO
| BGFX_STENCIL_OP_PASS_Z_ZERO
);
```

so it would wipe the reserved clip bit along with the winding count. The winding passes themselves are fine (`INCR`/`DECR` never carry into bit 7 for realistic winding numbers); it is only the zeroing cover pass that cannot be made selective.

## Proposed design

Reserve stencil bit `0x80` as the clip mask and **re-render the clip path before each draw** rather than trying to preserve the bit across draws. Correctness over speed, which seems the right trade for canvas 2D.

1. **Clip pass.** Render the clip path's winding into the low bits, then a cover pass over its bounding box with `TEST_NOTEQUAL, REF=0, RMASK=0x7f`, `OP_PASS_Z_REPLACE` with `FUNC_REF(0x80)` and `OP_FAIL_S_ZERO`. Result: stencil is `0x80` inside the clip region and `0x00` outside.

2. **Draw pass.** The existing winding passes are unchanged. The cover pass changes its test to `TEST_GREATER` with `REF=0x80, RMASK=0xff`, which passes exactly when the clip bit is set **and** the winding count is non-zero. Its existing `OP_*_ZERO` ops stay as they are and wipe both the winding bits and the clip bit, which is fine given step 3.

3. **Re-render.** Because step 2 destroys the mask, the clip pass runs again before the next draw while a clip is active.

Work involved:

- `nanovg.h` / `nanovg.cpp` -- a public `nvgClipPath()` / `nvgResetClip()`, clip path storage in `NVGstate` (so `nvgSave`/`nvgRestore` rewind it, which also fixes the missing save/restore of the clip region), and a new call type.
- `nanovg_babylon.cpp` -- the new clip pass, plus updated stencil state on every draw path (`glnvg__fill`, `glnvg__convexFill`, `glnvg__stroke`, `glnvg__triangles`).
- `Context.cpp` -- `Clip()` uses it; `RoundRect` can then set `m_pathHasNonRect` and keep its corners; `m_isClipped` / `ResetPathState` emulation and the `FillRect` special case all disappear.

## Risk

This changes the stencil state of *every* nanovg draw, so it affects all five graphics backends, and only D3D11/D3D12/Vulkan can be verified locally -- Metal and OpenGL/ES would need CI or device coverage. The intersection of clipping with the filter stack (which renders through intermediate framebuffers) needs explicit testing too.

Happy to pick this up; filing it rather than rushing it, since it is a rendering-correctness change with a wide blast radius.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.