w3c / w3c/webcodecs

Hardware acceleration and fingerprinting

Open
#896 2 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
HTML
Stars
1.3k
Forks
194
Avg merge
1d 13h
Merged PRs (30d)
3

Description

I just filed a bug on Bugzilla but what Firefox is doing is spec compliant, so I wanted to raise the issue here too.

Background

I'm making a website that heavily utilizes WebCodecs. I need to determine the encoding codec and always want to use hardware encoding (when available) as battery life is more important than fidelity (conference call).

  1. Loop through all codecs in order of network efficiency: av1, vp9, hevc, h254, vp8, etc
  2. Check if the browser supports hardware acceleration, if so return it.
  3. Repeat the loop but this time with software acceleration.

I'm using the hardwareAcceleration: "prefer-hardware" field to determine if hardware acceleration is supported. Here's an example of one of the checks:

VideoEncoder.isConfigSupported({
    codec: "av01.0.08M.08",
    width: 1280,
    height: 720,
    hardwareAcceleration: "prefer-hardware",
}).then(console.log)

Here's a little modal I created to visualize the process and the results on my M2 laptop and Chrome.

Image

Problem

Here's what Firefox reports:

Image

Ignore the partial support for video capture (MediaStreamTrackProcessor polyfill).

The problem is that Firefox is reporting hardware acceleration for AV1, VP9, etc. So my encoder will naively select one of those codecs and suddenly my CPU is on fire.

My guess is that Firefox is ignoring the hardwareAcceleration field to avoid fingerprinting. Here's the relevant section in the specification:

If a User Agent does not implement [media-capabilities] for reasons of fingerprinting, they SHOULD ignore the HardwareAcceleration preference

Proposed

If the browser won't respect hardwareAcceleration because it can be used for fingerprinting, then the browser should return an error somehow. I don't think the browser should silently ignore important configuration values without feedback.

If I knew that Firefox doesn't allow probing for hardware acceleration, then I would use H.264 because hardware acceleration is very common and easier on the CPU. Then I could also report hardware acceleration as "unknown" and maybe show a modal explaining why performance might be poor.

My suggestions:

  1. Don't return hardwareAcceleration in the config returned by isConfigSupported. Unfortunately, this isn't quite correct, as hardwareAcceleration is supported but only as a hint.
  2. Add "require-hardware" and "require-software" as options, throwing an exception (or returning false?) if they are used on browsers that care about fingerprinting.
  3. Firefox pls allow me to detect hardware acceleration. Otherwise I can't use newer codecs for fear of melting silicon.

I think #2 is the best option. It's kind of strange that Chrome returns isConfigSupported=false when "prefer-hardware" is used. If it's just a hint, like the "prefer" implies, then it should always return true if either software or hardware encoding is available... right?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the WebCodecs specification's hardware-acceleration section and the VideoEncoder.isConfigSupported example, then review the linked Bugzilla report and the two existing comments. The issue is done when the behavior for hardwareAcceleration under fingerprinting constraints and the observable result for authors are agreed and specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
api, audio-video-rtc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.