w3c / w3c/webcodecs

AudioDecoder should let you specify an format for resulting AudioData

Open
#859 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
HTML
Stars
1.3k
Forks
194
Avg merge
1d 13h
Merged PRs (30d)
3

Description

The User Agent is free to choose how the decoding will be performed and what format it will use when creating the AudioData.

In theory, how the data is stored should indeed be an implementation details.

However, what format is chosen will impact on how AudioData.copyTo will behave and how it will interpret the planeIndex parameter.
Additionally, there's a few WPTs that relies on the decoded AudioData to be interleaved and error otherwise (such as https://wpt.live/webcodecs/audio-encoder.https.any.js and its channelNumberVariationTests)

AudioData::copyTo
"The output AudioSampleFormat for the destination data. If not provided, the resulting copy will use this AudioData’s [[format]"

With the default option, if we were to call copyTo with a planeIndex value of 0 and the AudioData was interleaved, we would copy all planes.
While if the AudioData was planar, copyTo with a planeIndex of 0 would copy only that particular plane.

That the internal UA choice on what to use impact the default behaviour for later created objects and change the behaviour means that it cannot be considered an implementation detail, it's not transparent and is directly observable.
And real-life experience already shows web compatibility between Chrome and Firefox (such as https://w3c.github.io/webcodecs/samples/audio-video-player/audio_video_player.html where the code expects the decoded AudioData to be planar and will have an exception as it attempts to copy the planar data in an array of float) as it doesn't specify what format to use with copyTo)

So we should specify what the default format the AudioDecoder should use and provide the ability to override it in the configuration.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the WebCodecs AudioDecoder and AudioData definitions, especially AudioData.copyTo, then inspect the cited audio-encoder WPT and its channelNumberVariationTests. Done means the specification defines a default decoded format and a configuration override, with the affected tests behaving consistently.

Written by the indexing model from the issue text.

Assessment

Domain
api, audio-video-rtc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.