Comfy-Org / Comfy-Org/ComfyUI

Stable Audio 3 VAE outputs broadband noise: encode→decode round-trip destroys audio (MPS and CPU)

Open
#16,087 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Summary

The Stable Audio 3 VAE produces broadband noise on Apple Silicon (MPS) and on CPU. A plain **encode → decode round-trip of a known-good WAV** — no diffusion, no text conditioning — destroys the audio. The same round-trip through the Stable Audio Open 1.0 VAE is clean, and the same SA3 weights generate correct audio through Stability AI's official `stable-audio-3` package on the same machine.

### Minimal reproduction (no sampling involved)

Workflow: `CheckpointLoaderSimple` → `LoadAudio` → `VAEEncodeAudio` → `VAEDecodeAudio` → `SaveAudio`

- checkpoint: `stable_audio_3_small_sfx.safetensors` (from `Comfy-Org/stable-audio-3`, sha256 `ed9cf1b6172f1a8c2921a9560c21109ff3239524563ced9dce6dcdef41e2f515`)
- input: any normal 44.1 kHz stereo WAV

Result: the output is broadband noise — energy spread evenly from DC to Nyquist for the entire duration, with no transient, no harmonics and no decay.

Control: the identical workflow with `stable-audio-open-1.0.safetensors` returns the input essentially unchanged.

Text-to-audio is affected the same way, which is expected if the decoder is the broken stage.

### What this is not

Each of the following was tested and ruled out:

| Suspected cause | Test | Result |
|---|---|---|
| Wrong graph | Compared verbatim against the official `audio_stable_audio_3_medium_base` template | Matches (`t5gemma_b_b_ul2` loaded via `CLIPLoader` with `type: stable_audio`) |
| Wrong sampler | Official settings `lcm` / `simple` / steps 50 / cfg 7 | No change |
| Prompt format | Official example prompts including the trailing `Length: X seconds` token | No change |
| Precision | `--force-fp32`, `--fp32-vae` | No change |
| Device | `--cpu` | No change (so this is not MPS-specific) |
| Partial loading of `weight_norm` layers | `--gpu-only --disable-smart-memory` | No change |
| Corrupt weights | sha256 of checkpoint and text encoder vs `Comfy-Org/stable-audio-3` | Exact match |
| Latent shape | SA3 is 256ch / 4096, `EmptyLatentAudio` emits 64ch / 2048 | `fix_empty_latent_channels()` rescales correctly; output duration matches the expected value |
| Model variant | `small_sfx` and `small_sfx_base` | Both fail identically |
| Stale checkout | 24 commits behind master | No audio/latent commits in the gap |

No `missing`/`unexpected key` warnings appear on load, and the log shows `Requested to load SAT5GemmaModel` and `Requested to load SA3AudioVAE`, so detection and weight loading look correct.

### Proof the weights are fine

The same `small-sfx` weights, on the same Apple Silicon machine, produce correct audio through Stability AI's own `stable-audio-3` inference package (MPS, 8 steps, cfg 1.0): clear transient, harmonic structure, decay to silence.

### Where I would look

`SoftNormBottleneck` in `comfy/ldm/audio/vae_sa3.py` applies per-channel `scaling_factor` and `running_std` normalisation, while `latent_formats.StableAudio3` in `comfy/latent_formats.py` only declares `latent_channels` / `latent_dimensions` / `temporal_downscale_ratio` and carries no `process_in` / `process_out` scaling. If those two do not agree, decoded output would look exactly like this. I have not confirmed this — it is a starting point, not a diagnosis.

### Environment

- ComfyUI 0.34.0, commit `0a33ed6c`
- macOS 26.6.2, Apple M4, 16 GB unified memory, PyTorch MPS backend
- Python 3.12
- Also reproduced with `--cpu` on the same machine

Contributor guide

Open the contributing guide

Research direction

Reproduce the encode→decode workflow with stable_audio_3_small_sfx.safetensors, then compare it with stable-audio-open-1.0. Read SoftNormBottleneck in comfy/ldm/audio/vae_sa3.py alongside latent_formats.StableAudio3 in comfy/latent_formats.py, focusing on the scaling information mentioned in the report. Done means the Stable Audio 3 round-trip preserves the known-good WAV instead of producing broadband noise on CPU and MPS.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.