w3c / w3c/webcodecs

Clarify format of `new VideoFrame(CanvasImageSource)`

Open
#243 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

extension
Dominant language
HTML
Stars
1.3k
Forks
194
Avg merge
1d 13h
Merged PRs (30d)
3

Description

When creating a VideoFrame from a CanvasImageSource, the most obvious choice of format is an RGBA one. However, if the frame is going to be read back and encoded with 4:2:0 subsampling, creating a 4:2:0 frame in the first places can save half of the memory bandwidth.

I would propose that we explicitly allow implementations to choose the format of VideoFrames created in this way. In the future we will likely want to add a hint. This could just be the same format enum, but an implementation could interpret I420 as a hint for 4:2:0 and choose to create an NV12 frame. Alternatively the hint could be just in the form of a subsample resolution if that is more clear.

Under this proposal, Chrome's implementation might in some cases use a 4:2:0 format for accelerated CanvasImageSources when no hint is provided.

We could be more strict, but I suspect it's rarely better to fail or do a much more expensive operation. A VideoFrame conversion API is better suited to precise format transformations.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the VideoFrame(CanvasImageSource) constructor and review the existing issue discussion. Define whether implementations may choose the format, how an optional format or subsampling hint would work, and what behavior should be specified for accelerated CanvasImageSources. The proposal is complete when the format-selection semantics and compatibility expectations are precise.

Written by the indexing model from the issue text.

Assessment

Tech stack
html
Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.