NVIDIA / NVIDIA/TensorRT-Model-Connect
Feature: preserve multichannel audio in public results and WAV output
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 254
- Forks
- 58
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 235
Description
Motivation
While working on HeartMuLa support (#1227), I noticed that its 48 kHz stereo output cannot currently be represented unambiguously by the public audio result API. Could we add a small, mono-compatible multichannel output contract before wiring up the full codec output?
Current TRTMC interface
At a50cf5dc, AudioResult contains samples, num_samples, and sample_rate, but no channel count or layout. AudioChunkCallback likewise has no channel information. The CLI write_wav hardcodes num_channels = 1.
The sample rate is already configurable; the gap is preserving channels and defining sample-count semantics. Flattening stereo into the existing mono contract would not preserve the intended audio.
Proposed scope
- Add channel metadata to audio results, defaulting to mono, and document a canonical layout (for example, interleaved float samples).
- Explicitly distinguish total scalar samples from frames per channel, retaining existing mono behavior.
- Update WAV output to derive channel count, block alignment, byte rate, and data size consistently; validate malformed buffers. Audit reader behavior and document any intentional mono downmix separately.
- Decide how streaming can carry the same information without silently breaking existing callback users. A new callback type or overload may be preferable to changing the existing signature.
- Add CPU tests for unchanged mono output and stereo round trips with distinct left/right signals, including duration, sample rate, channel order, and invalid buffer lengths.
Initial implementation
PR #1256 proposes the complete-result and WAV-output portion, including channel-aware benchmark duration and CPU regression tests. It does not implement multichannel streaming, additional encoded formats, or an HTTP audio API.
Follow-up capabilities for discussion
- Multichannel streaming output: carry channel count, sample rate, and layout with chunks; define frame boundaries and end-of-stream behavior while preserving existing mono callback users. Test that concatenated chunks preserve channel order, sample count, and duration.
- Additional audio output formats: consider FLAC, MP3, and Opus alongside WAV, with explicit format selection, channel preservation, and clear errors when an encoder is unavailable. Agree on optional dependencies and codec limitations before implementation; lossy formats should not use byte-exact waveform comparisons as acceptance criteria.
- HTTP audio serving: discuss whether this belongs in TRTMC's serving layer and how callers should request audio generation and receive encoded audio with the correct content type, including streaming where supported. Endpoint design, request validation, cancellation, and error propagation need a separate scoped proposal.
These are proposed follow-ups, not implemented features or requirements to expand #1256. Would maintainers prefer separate tracking issues/PRs for them? I am happy to work on the agreed scope, starting with the output contract needed by #1227.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with core/runtime/include/trtmc/task.h and apps/cli/io.cpp, then review PR #1256 and its CPU regression tests. Check the proposed result metadata, sample-count semantics, WAV calculations, malformed-buffer validation, and the unresolved streaming callback design; done means the agreed scope is implemented without expanding into codecs or HTTP serving.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- audio-video-rtc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100