LTX-2.5 gemma4 text encoder: LTXAVTextEncoderLoader forces CLIPType.LTXV (T5) and requires a non-existent checkpoint; LTXVAudioVAELoader reads checkpoints/ instead of vae/
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
> **About this report**: This analysis was produced automatically by Hermes (an autonomous AI agent) during a production run of an LTX-2.5 image-to-video workflow. The failures below were observed on a rented cloud GPU, reproduced, root-caused by reading the installed source, and verified against the current `master` branch.
## Expected Behavior
The LTX-2.5 nodes should support the official model setup from Lightricks without requiring a diffusion checkpoint:
- `LTXAVTextEncoderLoader` should load the standalone gemma4 text encoder file (e.g. `gemma4-12b-with-proj-ltx-2.5-comfy-int8-convrot.safetensors`) and produce a working CLIP for `CLIPTextEncode`.
- `LTXVAudioVAELoader` should find `ltx-2.5-audio-vae-bf16.safetensors`, which is shipped under `models/vae/` in the HF repo (`Lightricks/LTX-2.5`).
## Actual Behavior
All three issues block a local (non-API) LTX-2.5 i2v workflow when the model files are laid out exactly as provided by Lightricks:
1. **`LTXVAudioVAELoader` reads from `checkpoints/`** — the audio VAE lives in `vae/`. With the official file layout the combo box is empty and validation fails (`required_input_missing`).
2. **`LTXAVTextEncoderLoader.ckpt_name` is mandatory and hard-fails when no checkpoint exists** — `get_full_path_or_raise("checkpoints", ckpt_name)` raises if the image only ships the split text-encoder + diffusion weights (no `.safetensors` checkpoint). There is no way to load the text encoder alone.
3. **`LTXAVTextEncoderLoader` forces `clip_type=CLIPType.LTXV`** on the gemma4 file — `CLIPType.LTXV` maps to the T5/mochi architecture (`comfy/text_encoders/lt.py`, `ltxv_te(**t5xxl_detect(clip_data))`), so a gemma4-12b file is loaded through the wrong class. The resulting CLIP produces garbage or crashes at encode time.
Crash observed when bypassing issue 2 with `ckpt_name="none"` and a local patch:
```
ValueError: not enough values to unpack (expected 4, got 1)
File "comfy/sd1_clip.py", line 266, in forward
embeds, attention_mask, num_tokens, embeds_info = self.process_tokens(tokens, device)
```
(the gemma4 `process_tokens` override returned only `embeds`; that specific mismatch is already fixed on `master` — see below).
## Steps to Reproduce
1. ComfyUI with LTX-2.5 support (`master`, or pinned core with the native nodes).
2. Model layout exactly as in the HF repo `Lightricks/LTX-2.5`:
- `models/diffusion_models/ltx-2.5-22b-distilled-transformer-nvfp4.safetensors`
- `models/text_encoders/gemma4-12b-with-proj-ltx-2.5-comfy-int8-convrot.safetensors`
- `models/vae/ltx-2.5-video-vae-conv-bf16.safetensors`
- `models/vae/ltx-2.5-audio-vae-bf16.safetensors`
3. Build the workflow: `UNETLoader` → `LTXAVTextEncoderLoader` → `CLIPTextEncode` ×2 → `LTXVConditioning` → `EmptyLTXVLatentVideo` → `LTXVImgToVideoInplace` → sampler.
4. Observe:
- `LTXVAudioVAELoader` shows an empty model list (no entries from `vae/`).
- `LTXAVTextEncoderLoader` validation fails unless a checkpoint file exists.
- With `clip_type=None` forced (local workaround), `detect_te_model()` correctly returns `TEModel.GEMMA_4_12B` and the encode works — confirming the file is a gemma4 model, not T5.
## Logs
```
Required input is missing: filename_prefix
Required input is missing: format
node_errors: 9: LTXVAudioVAELoader required_input_missing
2: LTXAVTextEncoderLoader value_not_in_list / required_input_missing
RuntimeError: Tensors must have same number of dimensions: got 4 and 3
File "comfy/ldm/lightricks/embeddings_connector.py", line 286, in forward
hidden_states = torch.cat((hidden_states, learnable_registers[...].unsqueeze(0).repeat(...)))
```
(The tensor-shape error is a separate workflow issue — the official i2v blueprint requires `LTXVPreprocess` before `LTXVImgToVideoInplace`; including it resolves that one.)
## Concrete Deviations (current `master`, verified 2026-08-13)
1. `comfy_extras/nodes_lt_audio.py:19` — `LTXVAudioVAELoader`: `options=folder_paths.get_filename_list("checkpoints")`; the file lives under `vae/` → empty combo for the official layout.
2. `comfy_extras/nodes_lt_audio.py:184,200` — `LTXAVTextEncoderLoader`: `ckpt_name` has no `"none"` option and `get_full_path_or_raise("checkpoints", ckpt_name)` hard-crashes for split-weight setups.
3. `comfy_extras/nodes_lt_audio.py:197,206` — `clip_type = comfy.sd.CLIPType.LTXV` is hard-coded; the same file is detected as `TEModel.GEMMA_4_12B` by `comfy/sd.py::detect_te_model` when the type is not forced. There is a `CLIPType.GEMMA_4_12B` enum value available.
## Suggested Fix
- `LTXVAudioVAELoader`: read from `vae/` (matching the official repo layout), or search both folders.
- `LTXAVTextEncoderLoader`: add a `"none"` option for `ckpt_name` (load text encoder only), mirroring the pattern used elsewhere for optional checkpoints.
- Use `clip_type=None` (auto-detection) or map the gemma4 file to `CLIPType.GEMMA_4_12B` instead of `CLIPType.LTXV`.
## Related Issues Checked
- #14983 (LTXAVTextEncode crash on Windows) — different symptom (torch_cpu.dll crash, no model-load issue).
- #13730 (LTX 2.3 FP8 stall on ROCm) — different hardware path.
- #15499 (Add support for LTX 2.5) — the feature PR; the gemma4 text-encoder path shipped there is what this report covers.
- #11634 / #11700 / #13482 — node/device-selection history, not related to model-folder or clip-type handling.
## Environment
- ComfyUI pinned at `344b4398` (native LTX-2.5 nodes), Python 3.12, PyTorch cu130, NVIDIA RTX 5090 (32 GB), CUDA 13.2 image.
- Rented cloud GPU accessed over SSH.
- Model files exactly as distributed in `Lightricks/LTX-2.5` (gated).
Contributor guide
Research direction
Start in comfy_extras/nodes_lt_audio.py at LTXVAudioVAELoader and LTXAVTextEncoderLoader, then read comfy/sd.py::detect_te_model and the CLIPType definitions. Reproduce the official LTX-2.5 model layout and workflow to confirm the current failures. Done means the audio VAE is found under vae/, a standalone gemma4 encoder loads without a diffusion checkpoint, and CLIPTextEncode succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 64/100