NVIDIA / NVIDIA/TensorRT-Model-Connect

Feature: preserve multichannel audio in public results and WAV output

Open
#1,254 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
254
Forks
58
Avg merge
1d 7h
Merged PRs (30d)
235

Description

Motivation

While working on HeartMuLa support (#1227), I noticed that its 48 kHz stereo output cannot currently be represented unambiguously by the public audio result API. Could we add a small, mono-compatible multichannel output contract before wiring up the full codec output?

Current TRTMC interface

At a50cf5dc, AudioResult contains samples, num_samples, and sample_rate, but no channel count or layout. AudioChunkCallback likewise has no channel information. The CLI write_wav hardcodes num_channels = 1.

The sample rate is already configurable; the gap is preserving channels and defining sample-count semantics. Flattening stereo into the existing mono contract would not preserve the intended audio.

Proposed scope
  1. Add channel metadata to audio results, defaulting to mono, and document a canonical layout (for example, interleaved float samples).
  2. Explicitly distinguish total scalar samples from frames per channel, retaining existing mono behavior.
  3. Update WAV output to derive channel count, block alignment, byte rate, and data size consistently; validate malformed buffers. Audit reader behavior and document any intentional mono downmix separately.
  4. Decide how streaming can carry the same information without silently breaking existing callback users. A new callback type or overload may be preferable to changing the existing signature.
  5. Add CPU tests for unchanged mono output and stereo round trips with distinct left/right signals, including duration, sample rate, channel order, and invalid buffer lengths.
Initial implementation

PR #1256 proposes the complete-result and WAV-output portion, including channel-aware benchmark duration and CPU regression tests. It does not implement multichannel streaming, additional encoded formats, or an HTTP audio API.

Follow-up capabilities for discussion
  • Multichannel streaming output: carry channel count, sample rate, and layout with chunks; define frame boundaries and end-of-stream behavior while preserving existing mono callback users. Test that concatenated chunks preserve channel order, sample count, and duration.
  • Additional audio output formats: consider FLAC, MP3, and Opus alongside WAV, with explicit format selection, channel preservation, and clear errors when an encoder is unavailable. Agree on optional dependencies and codec limitations before implementation; lossy formats should not use byte-exact waveform comparisons as acceptance criteria.
  • HTTP audio serving: discuss whether this belongs in TRTMC's serving layer and how callers should request audio generation and receive encoded audio with the correct content type, including streaming where supported. Endpoint design, request validation, cancellation, and error propagation need a separate scoped proposal.

These are proposed follow-ups, not implemented features or requirements to expand #1256. Would maintainers prefer separate tracking issues/PRs for them? I am happy to work on the agreed scope, starting with the output contract needed by #1227.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with core/runtime/include/trtmc/task.h and apps/cli/io.cpp, then review PR #1256 and its CPU regression tests. Check the proposed result metadata, sample-count semantics, WAV calculations, malformed-buffer validation, and the unresolved streaming callback design; done means the agreed scope is implemented without expanding into codecs or HTTP serving.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
audio-video-rtc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.