NVIDIA / NVIDIA/NeMo-Retriever
[FEA]: Add utilities to the API that allow for stripping audio tracks from common video formats
@ChrisJar is already working on this.
Since Jul 25, 2025.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 116
Description
Is this a new feature, an improvement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
Significant improvement
Please provide a clear description of problem this feature solves
As part of planning for video ingestion, there are a number of use cases we want to support, while minimizing the movement of data unnecessarily. In some cases we may only be interested in the audio component of a video file and would like to strip that data out as early as possible.
To support this, it would be good to have a set of utility functions in the API, and some initial prototyping to allow for converting files from their movie format into audio.
Describe the feature, and optionally a solution or implementation and any alternatives
These utilities should:
- Extract audio from a video file to a standard format (e.g.,
.wav,.mp3) - Support mono, stereo, or multi-channel configurations
- Optionally downmix to mono
- Allow specifying output sampling rate and bit depth
- Avoid full decode if passthrough is possible (e.g., extracting AAC directly)
Sample function signature.
def extract_audio_from_video(
video_path: str,
output_path: str,
audio_format: str = "wav",
stream_index: int | None = None,
downmix: bool = False,
sample_rate: int | None = None,
bit_depth: int | None = None,
) -> None:
"""
Extracts audio from a video file.
Parameters
----------
video_path : str
Path to the input video file.
output_path : str
Destination path for the extracted audio.
audio_format : str, optional
Desired output format (e.g., 'wav', 'mp3'), by default 'wav'.
stream_index : int, optional
Index of the audio stream to extract, if multiple streams are present.
downmix : bool, optional
If True, convert audio to mono.
sample_rate : int, optional
Resample output to this frequency (e.g., 16000, 44100).
bit_depth : int, optional
Set bit depth (e.g., 16, 24) if supported by format.
"""
...
def list_audio_streams(video_path: str) -> list[dict]:
"""
Lists audio streams present in a video file.
Parameters
----------
video_path : str
Path to the video file.
Returns
-------
list of dict
Metadata about each audio stream, e.g., codec, channels, sample rate.
"""
...
Example usage
from nv_ingest_api.util.media.audio_utils import extract_audio_from_video, list_audio_streams
# Inspect available audio streams
streams = list_audio_streams("my_video.mp4")
for i, stream in enumerate(streams):
print(f"Stream {i}: {stream['codec_name']}, {stream['channels']}ch")
# Extract stereo WAV
extract_audio_from_video(
video_path="my_video.mp4",
output_path="out_stereo.wav"
)
# Extract and downmix to mono MP3
extract_audio_from_video(
video_path="my_video.mp4",
output_path="out_mono.mp3",
audio_format="mp3",
downmix=True
)
# Extract second stream with resampling and custom bit depth
extract_audio_from_video(
video_path="my_multilang_video.mkv",
output_path="out_highres.wav",
stream_index=1,
sample_rate=48000,
bit_depth=24
)
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.