livepeer / livepeer/lpms

Audio clicking on segmented transcodes

Open
#152 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
283
Forks
72
Avg merge
27m
Merged PRs (30d)
1

Description

When transcoding audio with consecutive segments, harmonic content such as sine waves produces audible clicking when playing back transcoded results. This clicking shows up as a transient in spectrograms. Presumably this is because most audio encoders operate with a sliding window, which leads to discontinuities at the edges of each group of samples.

### Mitigation

One way to mitigate this effect is to pad the source segment with audio samples from adjacent segments. After transcoding, the padded samples can be dropped. However, it is possible that the efficiency of this depends on a few factors:

* The sample rate being constant between source and transcoded segments. If there is a change in sample rate (especially a downsample), it is possible there will be overlap with earlier segments.
* The required padding is understood. For AAC, 1024 samples at each end of the segment seems to be enough. However, may depend on the codec or the configuration of the codec. (Alternatively, three entire segments could be transcoded at once (prev/current/next) but this increases costs.)

This may have other implications in a live transcoding context, such as the need to wait for trailing padding to become available before submitting the source segment for transcoding.

Example of padding the end of a segment:
```bash
function pad {
frames=1
fname=$(basename $1)
ffmpeg -loglevel warning -hide_banner -y -i $1 -muxdelay 0 -vn -c:a copy -copyts out/audio.ts
ffmpeg -loglevel warning -hide_banner -y -i $2 -muxdelay 0 -vn -c:a copy -frames:a $frames -copyts out/end.ts
cat out/audio.ts out/end.ts > out/padded_$fname
}

pad in/source_504.ts in/source_505.ts
pad in/source_505.ts in/source_506.ts
pad in/source_506.ts in/source_507.ts
pad in/source_507.ts in/source_508.ts
```

Example of trimming the padding:
```bash
function trim {
fname=$(basename $1)
frames=$(ffprobe -loglevel warning -hide_banner -count_packets \
-select_streams a -show_streams $1 | grep nb_read_packets | grep -o '[0-9]*$')
frames=$(($frames-1))
ffmpeg -hide_banner -y -i $1 -muxdelay 0 -vn -c:a copy \
-frames:a $frames -copyts out/stripped_$fname
}

trim out/padded_source_504.ts
trim out/padded_source_505.ts
trim out/padded_source_506.ts
trim out/padded_source_507.ts
```

### Samples

Source: `ffmpeg -f lavfi -i sine -c:a aac -f hls -hls_time 2 -t 10 test.m3u8`

Source audio, concatenated:
```
$ cat test0.ts test1.ts test2.ts test3.ts test4.ts test5.ts > source.ts
```
![image](https://user-images.githubusercontent.com/292510/64314607-72ba4500-cf64-11e9-8ee3-a18769af187b.png)

Sample program to transcode audio:
```golang
// test.go
package main

import (
"fmt"
"os"

"github.com/livepeer/lpms/ffmpeg"
)

func main() {
inp := os.Args[1]
pfx := os.Args[2]
ffmpeg.InitFFmpeg()
for i := 0; i <= 5; i++ {
in := &ffmpeg.TranscodeOptionsIn{
Fname: fmt.Sprintf("%s%d.ts", inp, i),
}
out := []ffmpeg.TranscodeOptions{ffmpeg.TranscodeOptions{
Oname: fmt.Sprintf("%s%d.ts", pfx, i),
VideoEncoder: ffmpeg.ComponentOpts{Name: "drop"},
}}
_, err := ffmpeg.Transcode3(in, out)
if err != nil {
fmt.Println("Could not transcode ", err)
}
}
}
```
```
$ go run test.go foo transcoded
$ cat transcoded0.ts transcoded1.ts transcoded2.ts transcoded3.ts transcoded4.ts transcoded5.ts > transcoded.ts
```
Transcoded audio, concatenated:
![image](https://user-images.githubusercontent.com/292510/64314598-6635ec80-cf64-11e9-903c-7d141a7fb5ca.png)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the clicking with the provided ffmpeg HLS commands and test.go, then compare the concatenated source.ts and transcoded.ts spectrograms. Investigate the segment handling around ffmpeg.Transcode3 and the proposed padding and trimming commands. Done means segmented transcoding no longer produces the reported boundary transients without breaking the sample-rate and live-transcoding cases described.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
audio-video-rtc
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.