Comfy-Org / Comfy-Org/ComfyUI

LTXAV: freqs_cis_matrix crashes on the empty-audio branch — reshape(B, 0, heads, -1) is ambiguous (regression in 0.29.0 / #15056)

Open Beginner friendly
#15,233 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

## Summary

`LTXAVModel` cannot forward a **video-only** latent since v0.29.0. Any path that reaches the model without an audio slot dies in `freqs_cis_matrix`:

```
RuntimeError: cannot reshape tensor of 0 elements into shape [2, 0, 32, -1]
because the unspecified dimension size -1 can be any value and is ambiguous
```

The empty-audio branch is deliberate and still present in the code — `LTXAVModel.separate_audio_and_video_latents` (`comfy/ldm/lightricks/av_model.py:634`) fabricates a **zero-length** audio tensor when the latent has only one slot:

```python
vx = x[0]
ax = x[1] if len(x) > 1 else torch.zeros(
(vx.shape[0], self.num_audio_channels, 0, self.audio_frequency_bins),
device=vx.device, dtype=vx.dtype
)
```

That `0` propagates to `a_latent_coords`, so `_prepare_positional_embeddings` (`av_model.py:854`) calls `_precompute_freqs_cis` with a zero-length sequence and `freqs_cis_matrix` reshapes a 0-element tensor with `-1`.

## Minimal repro

No models or workflow needed — it's a pure function:

```python
import torch
from comfy.ldm.lightricks.model import freqs_cis_matrix

freqs_cis_matrix(torch.zeros(2, 0, 128), 0, True, 32, torch.float32)
# RuntimeError: cannot reshape tensor of 0 elements into shape [2, 0, 32, -1]
```

`torch.zeros(2, 7, 128)` works fine — the failure is specific to a 0-length sequence.

## Root cause

`7c59a078d` — *"Use comfy kitchen rope functions in ltx models" (#15056)*, first released in **v0.29.0**.

The old `split_freqs_cis` computed the last dimension **explicitly**:

```python
B, T, half_HD = cos_freq.shape
cos_freq = cos_freq.reshape(B, T, num_attention_heads, half_HD // num_attention_heads)
```

An explicit dim reshapes a 0-element tensor happily. The replacement uses `-1` (`comfy/ldm/lightricks/model.py:674-676`), which PyTorch cannot resolve when the tensor has no elements. So this worked through 0.28.x and hard-fails from 0.29.0 on.

## Proposed fix

Restore the explicit dim — identical result for `T > 0`, so it's a pure restore, not a behaviour change:

```python
B, T, HD = cos_freq.shape
head_dim = HD // num_attention_heads
cos_freq = cos_freq.reshape(B, T, num_attention_heads, head_dim)
sin_freq = sin_freq.reshape(B, T, num_attention_heads, head_dim)
```

Verified locally on 0.29.2 across `T=0` and `T>0`, split and non-split (padded) paths — shapes match the pre-patch output exactly wherever the pre-patch code ran at all.

I'm happy to open a PR if that's useful.

## How it shows up in practice

Core doesn't hit this itself — video-only checkpoints use `LTXVModel`, and the AV sampling paths always pack an audio latent — which is presumably why the branch went untested. It surfaces as soon as anything drives the AV model with a plain video latent. In my case that's a spatially-tiled sampler doing an LTX-2.3 upscale pass: the tile carrying the audio wrapper samples fine, and the next tile — plain video — crashes.

Note that `freqs_cis_matrix` is only the *first* zero-length site; I haven't yet confirmed the audio blocks and AV cross-attention downstream are all 0-length-safe on 0.29.x. If the intent is that the empty-audio branch should work, it may be worth a test that forwards a video-only latent end-to-end through `LTXAVModel`. If the intent is that it should *never* be reached, the fallback in `separate_audio_and_video_latents` would be clearer as an explicit error.

Possibly related, different bug: #13692 (LTXAV nested-latent / non-nested-noise mismatch).

## Environment

- ComfyUI **v0.29.2** (also present on 0.29.0/0.29.1 by inspection; absent on 0.28.x)
- Windows 11, Python 3.13.11 (portable/embedded), torch 2.11.0+cu130
- NVIDIA RTX PRO 6000 Blackwell, `--use-sage-attention`
- Model: LTX-2.3 AV

Contributor guide

Open the contributing guide

Research direction

Start with freqs_cis_matrix in comfy/ldm/lightricks/model.py and run the minimal zero-length repro from the issue, then compare the non-empty, split, and padded paths. Check how the empty-audio tensor from LTXAVModel.separate_audio_and_video_latents and _prepare_positional_embeddings in av_model.py reaches it. Done means zero-length and nonzero inputs produce the expected shapes without changing existing behavior; confirm whether downstream empty-audio paths need separate coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.