w3c / w3c/webcodecs

Could ImageDecoder support non-RGBA8 color formats?

Open
#677 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Question
Dominant language
HTML
Stars
1.3k
Forks
194
Avg merge
1d 13h
Merged PRs (30d)
3

Description

Is this the correct place to post feedback on the API standard?

WebCodecs seems to primarily exist to support videoconference applications. But, I am interested in it for a particular reason: I am creating WebGPU apps, and I could use ImageDecoder to load images into memory before uploading them as textures to the GPU. (You can have the browser load an image and send it directly to the GPU using ImageBitmap/copyExternalImageToTexture, and this is actually more efficient than routing through ImageDecoder, but imagine there is some sort of CPU-side preprocessing I want to do before creating the texture.)

ImageDecoder.decode produces a VideoFrame, which can copyTo pixel data into a buffer. The pixel data is formatted according to the VideoFrame's VideoPixelFormat. However, aside from RGB8 and RGBA8, the only other formats are video-centric color formats (YUV). Although ImageDecoder can be used to decode conventional image formats, it appears to be lacking support for some types color formats that browser image decoders handle without problems and which are relevant to my use case (WebGPU):

  • Fewer than three channels. PNG supports 1, 2, 3 or 4 channels. WebGPU supports one, two, or four-channel textures (R8, RG8). Although this is not really a problem since I could trivially take in the RGBA or RGBX data from VideoFrame and discard the channels I don't need, there will be a performance concern; for a grayscale PNG, you will be taking the time to duplicate the one channel into four, and then I will have to add an extra step to copy the one channel into another buffer.
  • More than 8 bits per color component. PNG supports 16-bit color channels, and so does WebGPU (WebGPU also supports some oddities such as 32-bit color channels and floating-point color channels, though I'm not sure if any web image format can convey this data).

For example, here is a PNG image containing one 16-bit grayscale channel (explanation here), and browsers (my copy of Chrome at least) are able to display this image fine. (I haven't investigated the color formats supported by webp, JPEGXL etc)

This might seem trivial because most computers can only display 8-bit color components anyway. But HDR10 displays are increasingly common, and WebGPU introduces a separate interesting use case (although again, with WebGPU you'd be more likely to use the ImageBitmap path) because you might create textures containing non-pixel data, IE, a texture might contain some sort of unusual material map which for some reason uses more than 8 bits of resolution, and that texture would be an input to a shader rather than being displayed directly. You could also imagine JavaScript applications that use PNG to compress non-image tabular data. At the moment, ImageDecoder is the closest thing that the browser has to a sensible "load image, return pixel data to CPU as an array" API, so it would be nice if it could support some of these diverse color formats that the browser itself can.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ImageDecoder.decode operation and the VideoPixelFormat enum described in the linked WebCodecs specification, then compare them with the linked WebGPU texture formats. Done would require an agreed API design and specification changes that expose relevant non-RGBA8 formats, including fewer channels and more than 8 bits per component.

Written by the indexing model from the issue text.

Assessment

Tech stack
html
Domain
api, computer-graphics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.