TorchAudio Dispatcher Migration

Overview

We propose the following end state for TorchAudio’s I/O functions info, load, save:

• FFmpeg is the primary backend for TorchAudio’s I/O functions info, load, save.
• FFmpeg-, SoX-, and soundfile-based backends are user-selectable from said I/O functions and are no longer determined by global state.
• All of FFmpeg, SoX, and soundfile are optional dependencies.
• I/O functions no longer support TorchScript.

Context

TorchAudio’s functions info, load, and save currently rely on two third-party libraries: SoX and soundfile. Whereas SoX is used in the Linux and Mac distributions, soundfile is used in the Windows distribution.

Through the years, we’ve encountered several issues with SoX:

• Its handling of in-memory decoding is buggy and requires a local patch to fix.
• It accesses the internal structure of FILE* object. As a result, 14.4.42 does not compile on Windows with MSVC newer than 2013. This precludes us from using SoX across platforms.
• It attempts to rewind stdin.
• It has not been actively developed/maintained since 2015, which doesn’t lend confidence that the aforementioned issues will be addressed.
• It has caused other user-facing problems:
  • https://github.com/pytorch/audio/issues/2870
  • https://github.com/pytorch/audio/issues/2356

Separately, our work around streaming I/O introduced FFmpeg as a dependency. FFmpeg's advantages over SoX include the following:

• It’s battle tested. It’s been developed for over 20 years now and is widely used in industry.
• The library code is portable across Linux, Mac, and Windows.
• It supports a wide variety of codecs, from basic to advanced, for both audio and video.
• It supports GPU acceleration in decoding and encoding.
• The C API offers a high degree of customizability.
  • It abstracts away many things like codecs, file formats, and devices.
  • It allows for implementing custom I/O such as in-memory decoding/encoding with file-like object protocol.
• It’s being actively developed, with the latest version (5.1) having been released in July 2022.

End state

To address the issues above, we propose the following end state:

• FFmpeg is the primary backend for TorchAudio’s I/O functions info, load, save.
• FFmpeg-, SoX-, and soundfile-based backends are user-selectable from said I/O functions and are no longer determined by global state.
• All of FFmpeg, SoX, and soundfile are optional dependencies.
• I/O functions no longer support TorchScript.

We anticipate this end state bringing greater cross-platform consistency, simplifying our codebase, and delivering an improved user experience.

Plan

Release 2.0

• Introduce option to {info, load, save} that allows users to choose any of FFmpeg, SoX, and soundfile as the I/O backend for both file paths and file objects, while preserving the existing behavior, i.e. Linux and Mac distributions default to relying on SoX, Windows distributions default to relying on soundfile.
  • Doing so naturally removes TorchScript support in {info, load, save}.
• Add deprecation warnings that convey that release 2.0 will make FFmpeg the default backend for files and file objects for {info, load, save} and encourage users to switch over to FFmpeg.

Release 2.1

• Make FFmpeg the default backend for files and file objects for {info, load, save} across all platforms. (#3241)
• Make SoX an optional dependency that is dynamically linked if available. Linking enables the SoX backend and torchaudio.sox_effects. #3497
• Remove file-like object handling for the SoX backend. #3035

Release 2.2

• Remove dependence of backend selection on global state. #3559

Review the info, load, and save entry points and the Release 2.0–2.2 checklist. All listed work is marked complete, so confirm whether any follow-up remains before starting. No file or test entry point is named; completion is defined by the proposed backend-selection end state.