w3c / w3c/webcodecs

Improve memory locality for higher performances and reduced memory working set

Open
#212 15 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

extension
Dominant language
HTML
Stars
1.3k
Forks
194
Avg merge
1d 13h
Merged PRs (30d)
3

Description

We can add a second parameter to decode/encode and the output callbacks (for audio, video and images), that would be respectively an output buffer or handle, and the input buffer or handle (and also mirror it for encode vs. decode), to allow authors to reuse memory and avoid lots of memory traffic. GCs are amazing, but it's always better to not GC, reuse allocations, and always use the same bit of the address space, especially in very demanding multimedia scenarios.

This is really compelling whenever the input (resp. output for decode) is regular memory, since it means the working set is going to be pretty tight (for example help with https://github.com/w3c/webtransport/issues/231), but is really nice even when some buffers are backed by (say) GPU memory (because you're not doing malloc/free pairs for input/output buffers for respectively decoding and encoding). I'm not sure if GPU-memory backed surfaces can work in the same way.

Essentially, the proposal is to "pass through" the input to the output so that it can be reused. This reduces allocator use, but more importantly greatly improves memory locality.

When not using this new parameter, the implementation allocates as usual. This means that authors don't have to try to guess the buffer size, it's allocated once and then can be resized. The buffer is released naturally because it isn't referenced, immediately. In short, this is not a breaking change, but allows developers that want to control their memory allocation to be able to do so (and I believe this is something that is important for quite a few domains).

When the provided buffer is too small, the browser can allocate a new buffer, or use more complex tricks (realloc, or maybe it finds that the array buffer backing memory storage was in fact big enough, etc.).

When the provided buffer is way too big, the implementation can decide to swap it for a buffer that has a more appropriate size.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names the decode and encode APIs and their audio, video, and image output callbacks, but no files, tests, or entry points. Start by locating those API definitions and determine how reusable input and output buffers should be specified; done means an agreed, specified design that covers sizing, replacement, and the unchanged default behavior.

Written by the indexing model from the issue text.

Assessment

Domain
api, audio-video-rtc, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.