LTXAV: freqs_cis_matrix crashes on the empty-audio branch — reshape(B, 0, heads, -1) is ambiguous (regression in 0.29.0 / #15056)
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
## Summary
`LTXAVModel` cannot forward a **video-only** latent since v0.29.0. Any path that reaches the model without an audio slot dies in `freqs_cis_matrix`:
```
RuntimeError: cannot reshape tensor of 0 elements into shape [2, 0, 32, -1]
because the unspecified dimension size -1 can be any value and is ambiguous
```
The empty-audio branch is deliberate and still present in the code — `LTXAVModel.separate_audio_and_video_latents` (`comfy/ldm/lightricks/av_model.py:634`) fabricates a **zero-length** audio tensor when the latent has only one slot:
```python
vx = x[0]
ax = x[1] if len(x) > 1 else torch.zeros(
(vx.shape[0], self.num_audio_channels, 0, self.audio_frequency_bins),
device=vx.device, dtype=vx.dtype
)
```
That `0` propagates to `a_latent_coords`, so `_prepare_positional_embeddings` (`av_model.py:854`) calls `_precompute_freqs_cis` with a zero-length sequence and `freqs_cis_matrix` reshapes a 0-element tensor with `-1`.
## Minimal repro
No models or workflow needed — it's a pure function:
```python
import torch
from comfy.ldm.lightricks.model import freqs_cis_matrix
freqs_cis_matrix(torch.zeros(2, 0, 128), 0, True, 32, torch.float32)
# RuntimeError: cannot reshape tensor of 0 elements into shape [2, 0, 32, -1]
```
`torch.zeros(2, 7, 128)` works fine — the failure is specific to a 0-length sequence.
## Root cause
`7c59a078d` — *"Use comfy kitchen rope functions in ltx models" (#15056)*, first released in **v0.29.0**.
The old `split_freqs_cis` computed the last dimension **explicitly**:
```python
B, T, half_HD = cos_freq.shape
cos_freq = cos_freq.reshape(B, T, num_attention_heads, half_HD // num_attention_heads)
```
An explicit dim reshapes a 0-element tensor happily. The replacement uses `-1` (`comfy/ldm/lightricks/model.py:674-676`), which PyTorch cannot resolve when the tensor has no elements. So this worked through 0.28.x and hard-fails from 0.29.0 on.
## Proposed fix
Restore the explicit dim — identical result for `T > 0`, so it's a pure restore, not a behaviour change:
```python
B, T, HD = cos_freq.shape
head_dim = HD // num_attention_heads
cos_freq = cos_freq.reshape(B, T, num_attention_heads, head_dim)
sin_freq = sin_freq.reshape(B, T, num_attention_heads, head_dim)
```
Verified locally on 0.29.2 across `T=0` and `T>0`, split and non-split (padded) paths — shapes match the pre-patch output exactly wherever the pre-patch code ran at all.
I'm happy to open a PR if that's useful.
## How it shows up in practice
Core doesn't hit this itself — video-only checkpoints use `LTXVModel`, and the AV sampling paths always pack an audio latent — which is presumably why the branch went untested. It surfaces as soon as anything drives the AV model with a plain video latent. In my case that's a spatially-tiled sampler doing an LTX-2.3 upscale pass: the tile carrying the audio wrapper samples fine, and the next tile — plain video — crashes.
Note that `freqs_cis_matrix` is only the *first* zero-length site; I haven't yet confirmed the audio blocks and AV cross-attention downstream are all 0-length-safe on 0.29.x. If the intent is that the empty-audio branch should work, it may be worth a test that forwards a video-only latent end-to-end through `LTXAVModel`. If the intent is that it should *never* be reached, the fallback in `separate_audio_and_video_latents` would be clearer as an explicit error.
Possibly related, different bug: #13692 (LTXAV nested-latent / non-nested-noise mismatch).
## Environment
- ComfyUI **v0.29.2** (also present on 0.29.0/0.29.1 by inspection; absent on 0.28.x)
- Windows 11, Python 3.13.11 (portable/embedded), torch 2.11.0+cu130
- NVIDIA RTX PRO 6000 Blackwell, `--use-sage-attention`
- Model: LTX-2.3 AV
Contributor guide
Research direction
Start with freqs_cis_matrix in comfy/ldm/lightricks/model.py and run the minimal zero-length repro from the issue, then compare the non-empty, split, and padded paths. Check how the empty-audio tensor from LTXAVModel.separate_audio_and_video_latents and _prepare_positional_embeddings in av_model.py reaches it. Done means zero-length and nonzero inputs produce the expected shapes without changing existing behavior; confirm whether downstream empty-audio paths need separate coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 75/100