google / google/ExoPlayer

Support low latency custom audio signal processing

Open
#8,970 7 comments 2 reactions 1 assignee Assigned to @tonihei View on GitHub
enhancement low priority
Dominant language
Java
Stars
21.9k
Forks
6k
PR merge metrics
No merged PRs in 30d

Description

Currently ExoPlayer plays using AudioTrack configured with a significant (>250ms) buffer. This is done to avoid underrun and works well for most of our users. Nevertheless, it prevents some use-cases that require low latency signal processing.

## Goal

Such high latency is an issue when modifying the media (usually the audio) depending on low latency information (eg spacialized audio must react to user head movement with as low latency as possible). This issue only consider the audio pipeline as (I believe) low latency video DSP can already be implemented using shaders.
This was requested in #8962, #8722, #8665 (and possibly other).

This issue is NOT about real time media playback (eg a video call). End to end low latency playback is not a goal of ExoPlayer. Making all of ExoPlayer pipelines low latency would need a deep redesign and negatively affect power efficiency.

## Solutions
To allow apps to inject custom Audio Digital Signal Processing (ADSP) in ExoPlayer and have minimal latency on the playback, the audio produced by the ADSP needs to be played to the user as fast as possible.

ExoPlayer post ADSP has very little internal buffer, so most of the latency occurs after the write to the AudioTrack.

There are 2 main way to reduce the latency of ExoPlayer post ADSP playback:
1) reduce the AudioTrack buffer size and switch to performance mode rather than power efficiency mode.
2) use Oboe to play instead of AudioTrack.

### AudioTrack tuning
Having a low latency AudioTrack configuration (solution 1) is far more simple to implement (see https://github.com/google/ExoPlayer/issues/8665#issuecomment-789595598) and should greatly improve the situation.

Nevertheless it will not allow latency to be as low as Oboe. As very rough estimate, I would guess an optimised AudioTrack (with fast mixer) could reach a 20ms. This is of course extremely dependent on the device (@dturner might have better estimates).

### Oboe
An Oboe extension on the other hand could reach the best possible latency with as low as >10ms on device supporting Aaudio MMAP NOIRQ.
Nevertheless, this would require significant work to interfacing between Oboe and ExoPlayer, tuning and maintenance, so it should only be done if the low enough latency can't be achieved with AudioTrack tuning.

## Adding latency before ADSP
Whatever solution is used, ExoPlayer current AudioTrack buffer was chosen because such big buffer allows smoothing the decoding speed jitter (decoders sometime decode in bursts). By having a low latency path between from the decoder output to the audio output, ExoPlayer runs very much at risks of underruning every time the decoder takes longer to decode than a packet time. This could manifests itself for example by having audio glitch whenever a cpu burst occurs (eg user scrolls a page).

To avoid that a buffer should be added between the audio decoder and the ADSP to leave the decoder->output latency unaffected by the low latency AudioSink:
- before: Decoder -> ADSP --[big buffer (AudioTrack)]--> Speaker
- after : Decoder --[big buffer (ExoPlayer)]--> ADSP -[low latency AudioTrack]-> Speaker

### Jitter smoothing buffer Implementation
This new buffer could take the form of an AudioProcessor to easily remove it from the processing chain, or it could be part of DefaultAudioSink.

## Additional information

Overview of ExoPlayer's audio pipeline: https://github.com/google/ExoPlayer/issues/8722#issuecomment-801925912
Previous discussion on minimising ADSP latency: https://github.com/google/ExoPlayer/issues/8665#issuecomment-789595598

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.