linebender / linebender/piet

Make piet capable of being a back-end for resvg

Open
#32 16 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
1.4k
Forks
98
PR merge metrics
No merged PRs in 30d

Description

The resvg project has a very helpful [list of requirements](https://github.com/RazrFalcon/resvg/blob/master/docs/backend_requirements.md) for back-ends. I think piet is now at a point where it would be possible to start coding such a back-end, and it makes sense to carefully review where we stand.

## Imaging model and blending/compositing

Right now, piet effectively implements a painter's imaging model, where each object is alpha-composited on top of pixels on the canvas surface. There is an entire class of operations - group opacity, nonstandard blend modes, using an image as a mask, and of course filters, that do not fit into this imaging model. I'm hesitating slightly at this point, because I want to make sure to expose this functionality in a way that piet back-ends can efficiently implement them, and also doesn't add too much complexity and burden for implementers.

The imaging model of resvg is fundamentally a tree, with nodes representing various blend and mask operations. The way resvg works with, say, the Cairo back-end, is to treat the graphics library as a source for pixel data, and do the blending and filter operations in software. We can do this approach, and it is probably the shortest path to getting a working integration with resvg. Basically it requires exposing an interface for creating an image surface, and for getting RGBA pixels out of the image surface. This is is already done to some extent in the demo apps for writing image data, but (significantly) not in the web back-end, as the demo app has a one-way flow of data, drawing on the screen.

When rendering is done on GPU, getting pixel data back is inefficient. Doing blending operations in software is doubly so, especially for pointwise operations (including alpha groups, mask, and blend modes), all of which can be implemented efficiently in GPU shaders. The Direct2D library has a rich set of [effects](https://docs.microsoft.com/en-us/windows/desktop/Direct2D/built-in-effects), all of which seem to be available on Windows 7 with platform update. *Not* using these is leaving performance on the table.

That said, requiring the complete set of effects for all piet back-ends may be onerous. Further, many of these effects are of fairly limited use in UI contexts. If I were to prioritize, I'd pick group opacity, as that is something that's quite useful for building UI, and is also reasonably easy to implement. I think it can serve as a bit of a proxy for potentially more sophisticated blend modes.

For blend modes, it's a bit confusing what functionality to support. resvg seems to implement a [limited set,](https://github.com/RazrFalcon/resvg/blob/master/usvg/src/tree/attributes.rs#L602) consisting of Normal, Multiply, Screen, Darken, and Lighten. The [SVG Compositing spec](https://www.w3.org/TR/SVGCompositing/) has a much larger set. It also specifies complex options such as "enable-background" and "knock-out". The latter is effectively a second alpha channel, and seems to be imported from PDF 1.4 transparency. The complexity of implementing that for the web back-end would be considerable, and doing so performantly is challenging. But I think expanding the set of "comp-op" values is fairly straightforward, as all 3 current back-ends seem to support the same basic set.

What I'm leaning towards is exposing a "get RGBA pixels" interface so resvg can use software techniques for blending, and punting on the complex blends, with the likely exception of group opacity.

## Sophisticated text

There's no question piet needs fancier text (partially captured in #8 and #10), and there are number of text things in the resvg requirements list. I would like to punt on this for now. Part of the issue is that the web canvas back-end has only primitive text, as does the *current* Cario "toy text" implementation. For web canvas, we'll have to return "not implemented" error results for some queries, or fake it somehow. For the others, I want to get a solid text solution, but feel this is orthogonal to most of the rest of the resvg requirements

## Gradients

To some extent, gradients are the easiest problem, though of course the devil is always in the details. I propose to subsume gradients under the existing `Brush` associated type, and simply add more builders. It is likely we'll have to do some workarounds for [rendering quirks](https://www.reddit.com/r/rust/comments/ackrl7/followups_on_2d_graphics_proposal/edb5d86/).

In any case, I'm posting this issue to get a read on whether it makes sense to start implementation work on integrating resvg and piet, and what needs to be done on the piet side. @RazrFalcon is of course invited to provide feedback as well.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with resvg's backend requirements document and compare them with piet's current demo image-output path, web back-end, and Brush API. Review the unresolved scope around RGBA pixel access, group opacity, blend modes, gradients, and text, including issues #8 and #10. Done requires an agreed piet-side integration scope and implementation plan; the issue names no specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, computer-graphics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.