NVIDIA / NVIDIA/NeMo-Retriever

[FEA]: Add utilities to the API that allow for stripping audio tracks from common video formats

Open
#916 1 comment 0 reactions 1 assignee View on GitHub

@ChrisJar is already working on this.

Since Jul 25, 2025.

feature request
Dominant language
Python
Stars
3k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Significant improvement

Please provide a clear description of problem this feature solves

As part of planning for video ingestion, there are a number of use cases we want to support, while minimizing the movement of data unnecessarily. In some cases we may only be interested in the audio component of a video file and would like to strip that data out as early as possible.

To support this, it would be good to have a set of utility functions in the API, and some initial prototyping to allow for converting files from their movie format into audio.

Describe the feature, and optionally a solution or implementation and any alternatives

These utilities should:

  • Extract audio from a video file to a standard format (e.g., .wav, .mp3)
  • Support mono, stereo, or multi-channel configurations
  • Optionally downmix to mono
  • Allow specifying output sampling rate and bit depth
  • Avoid full decode if passthrough is possible (e.g., extracting AAC directly)

Sample function signature.

def extract_audio_from_video(
    video_path: str,
    output_path: str,
    audio_format: str = "wav",
    stream_index: int | None = None,
    downmix: bool = False,
    sample_rate: int | None = None,
    bit_depth: int | None = None,
) -> None:
    """
    Extracts audio from a video file.

    Parameters
    ----------
    video_path : str
        Path to the input video file.
    output_path : str
        Destination path for the extracted audio.
    audio_format : str, optional
        Desired output format (e.g., 'wav', 'mp3'), by default 'wav'.
    stream_index : int, optional
        Index of the audio stream to extract, if multiple streams are present.
    downmix : bool, optional
        If True, convert audio to mono.
    sample_rate : int, optional
        Resample output to this frequency (e.g., 16000, 44100).
    bit_depth : int, optional
        Set bit depth (e.g., 16, 24) if supported by format.
    """
    ...


def list_audio_streams(video_path: str) -> list[dict]:
    """
    Lists audio streams present in a video file.

    Parameters
    ----------
    video_path : str
        Path to the video file.

    Returns
    -------
    list of dict
        Metadata about each audio stream, e.g., codec, channels, sample rate.
    """
    ...

Example usage

from nv_ingest_api.util.media.audio_utils import extract_audio_from_video, list_audio_streams

# Inspect available audio streams
streams = list_audio_streams("my_video.mp4")
for i, stream in enumerate(streams):
    print(f"Stream {i}: {stream['codec_name']}, {stream['channels']}ch")

# Extract stereo WAV
extract_audio_from_video(
    video_path="my_video.mp4",
    output_path="out_stereo.wav"
)

# Extract and downmix to mono MP3
extract_audio_from_video(
    video_path="my_video.mp4",
    output_path="out_mono.mp3",
    audio_format="mp3",
    downmix=True
)

# Extract second stream with resampling and custom bit depth
extract_audio_from_video(
    video_path="my_multilang_video.mkv",
    output_path="out_highres.wav",
    stream_index=1,
    sample_rate=48000,
    bit_depth=24
)
Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.