huggingface / huggingface/diffusers

`LTX2*Pipeline.from_single_file` cannot load LTX 2.3 (or 2.0) combined checkpoints: unmapped 2.3 keys leave meta tensors, and the LTX2 converters drain the shared checkpoint dict

Aperta
#14,145 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug models pipelines single-file
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

### Describe the bug

Pipeline-level `from_single_file` is broken for the Lightricks LTX-2 combined (flat) checkpoints. There are two independent bugs; both are present on current `main` (verified in source at commit `208704a`, 2026-07-08). **I intend to submit a pull request for this** — a minimal, verified fix is ready.

**Bug 1 — the LTX2 key converters don't map the 2.3-only key names, and unmapped keys fail only later, as meta tensors.**

`convert_ltx2_transformer_to_diffusers` (`loaders/single_file_utils.py`) does not map the 2.3 prompt-register modulation keys:

```
model.diffusion_model.prompt_adaln_single.* -> prompt_adaln.* (6 keys)
model.diffusion_model.audio_prompt_adaln_single.* -> audio_prompt_adaln.* (6 keys)
```

(`LTX2VideoTransformer3DModel` has had `prompt_adaln` / `audio_prompt_adaln` modules since #13217, but the converter was never extended.)

Likewise `convert_ltx2_vae_to_diffusers`' rename dict stops at the 2.0 decoder's `up_blocks.6`; the 2.3 video VAE decoder has one more up block:

```
vae.decoder.up_blocks.7.* -> decoder.up_blocks.3.upsamplers.0.* (2 keys)
vae.decoder.up_blocks.8.* -> decoder.up_blocks.3.* (16 keys)
```

Because single-file loading initializes the model under `init_empty_weights` and unconverted checkpoint keys are merely reported as "unexpected", the affected modules silently keep meta tensors. Nothing fails at load time — the error surfaces later, in `pipeline.to(device)`:

```
NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.
```

This bites even the model-level `LTX2VideoTransformer3DModel.from_single_file(...)`.

**Bug 2 — every LTX2 converter consumes the whole shared checkpoint dict, so only the first checkpoint-sourced component gets any weights.**

All three LTX2 converters (`convert_ltx2_transformer_to_diffusers`, `convert_ltx2_vae_to_diffusers`, `convert_ltx2_audio_vae_to_diffusers`) start with:

```python
converted_state_dict = {key: checkpoint.pop(key) for key in list(checkpoint.keys())}
```

In pipeline-level `from_single_file`, `load_single_file_sub_model` passes the **same** checkpoint dict to each component, and `FromOriginalModelMixin.from_single_file` calls `checkpoint_mapping_fn` on it without copying. The first component loaded (e.g. the transformer) therefore pops **all** keys — including `vae.*`, `audio_vae.*`, `vocoder.*` — and every later component receives an empty dict, failing with:

```
SingleFileComponentError: Failed to load AutoencoderKLLTX2Video. Weights for this component appear to be missing in the checkpoint.
```

This bug is version-independent: it affects LTX **2.0** combined checkpoints (e.g. `Lightricks/LTX-2` `ltx-2-19b-dev.safetensors`) the same way — as far as I can tell the pipeline-level single-file path for LTX2 has never worked (there are no LTX2 tests under `tests/single_file/`). The older `convert_ltx_vae_checkpoint_to_diffusers` (LTX-Video 0.9.x) already avoids this by slicing the dict by prefix (`... if "vae." in key`).

**Proposed fix** (contained to the three LTX2 converters in `loaders/single_file_utils.py`):

1. Add the 2.3 renames: transformer `prompt_adaln_single` → `prompt_adaln`, `audio_prompt_adaln_single` → `audio_prompt_adaln` (inert for 2.0, which lacks these keys); video VAE `up_blocks.7` → `up_blocks.3.upsamplers.0`, `up_blocks.8` → `up_blocks.3`.
2. Make each converter consume only its own prefix (`model.diffusion_model.` / `vae.` / `audio_vae.`), falling back to all keys when the prefix is absent (extracted single-component files) — mirroring `convert_ltx_vae_checkpoint_to_diffusers`.

Related follow-up (out of scope for the fix): `infer_diffusers_model_type` maps every LTX2 checkpoint to `ltx2-dev` → `Lightricks/LTX-2`, so a 2.3 checkpoint silently gets the 2.0 architecture config unless `config=` is passed explicitly. A `model.diffusion_model.prompt_adaln_single.*` key unambiguously identifies 2.3 and could drive a 2.3 default config.

cc @dg845 (author of #13217, which added LTX-2.3 support).

### Reproduction

```python
import torch
from diffusers import LTX2Pipeline

pipe = LTX2Pipeline.from_single_file(
"https://huggingface.co/Lightricks/LTX-2.3/blob/main/ltx-2.3-22b-distilled-1.1.safetensors",
config="diffusers/LTX-2.3-Distilled-Diffusers",
torch_dtype=torch.bfloat16,
)
pipe.to("cuda")
```

Depending on which component loads first you get either the `SingleFileComponentError` (bug 2) or, after working around bug 2, the meta-tensor `NotImplementedError` from `pipe.to()` (bug 1).

Model-level reproduction of bug 1 alone (no pipeline involved):

```python
import torch
from diffusers import LTX2VideoTransformer3DModel

transformer = LTX2VideoTransformer3DModel.from_single_file(
"https://huggingface.co/Lightricks/LTX-2.3/blob/main/ltx-2.3-22b-distilled-1.1.safetensors",
config="diffusers/LTX-2.3-Distilled-Diffusers",
subfolder="transformer",
torch_dtype=torch.bfloat16,
)
transformer.to("cuda") # NotImplementedError: Cannot copy out of meta tensor
# transformer.prompt_adaln.linear.weight.is_meta == True
```

**Key-level evidence (no GPU needed).** Running the three converters in sequence on one shared dict built from the real `ltx-2.3-22b-distilled-1.1.safetensors` header (dummy values; the converters only move keys), and diffing each result against the matching component of `diffusers/LTX-2.3-Distilled-Diffusers`:

Current `main`:

```
[transformer] converted=5689 expected=4186 missing=12 unexpected=1515
MISSING prompt_adaln.* / audio_prompt_adaln.* (12 keys)
UNEXPECTED *_single (unmapped) + vae.* + audio_vae.* + vocoder.* (swallowed from the other components)
[vae] converted=0 expected=170 missing=170 # shared dict already drained by the transformer converter
[audio_vae] converted=0 expected=102 missing=102 # shared dict already drained
```

`convert_ltx2_vae_to_diffusers` in isolation (only `vae.*` keys), showing the up-block gap:

```
missing: 18 unexpected: 18
UNEXPECTED decoder.up_blocks.7.*, decoder.up_blocks.8.*
MISSING decoder.up_blocks.3.upsamplers.0.*, decoder.up_blocks.3.resnets.*
```

With the proposed fix, all three components reach exact key parity for **both** generations:

```
LTX 2.3 (ltx-2.3-22b-distilled-1.1 vs diffusers/LTX-2.3-Distilled-Diffusers):
[transformer] 4186/4186 [vae] 170/170 [audio_vae] 102/102 (0 missing, 0 unexpected)
LTX 2.0 regression check (ltx-2-19b-dev vs Lightricks/LTX-2):
[transformer] 3510/3510 [vae] 184/184 [audio_vae] 102/102 (0 missing, 0 unexpected)
```

After conversion, only `vocoder.*` and `text_embedding_projection.*` remain in the shared dict — expected, as those are not single-file-loadable and come from the config repo.

### Logs

```shell
Meta-tensor failure from `pipe.to()` / `transformer.to()` (bug 1):

NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

Starved-component failure (bug 2), for whichever checkpoint-sourced component loads after the first:

SingleFileComponentError: Failed to load AutoencoderKLLTX2Video. Weights for this component appear to be missing in the checkpoint.
```

### System Info

- `diffusers` version: 0.39.0.dev0 (git main). The bug is also confirmed by reading the source on `main` at commit `208704a` (2026-07-08).
- Platform: Linux aarch64 (NVIDIA DGX Spark / GB10)
- Python: 3.12
- PyTorch: 2.x (cu130)
- `transformers`, `safetensors`, `huggingface_hub`: current releases
- Checkpoint: `Lightricks/LTX-2.3` → `ltx-2.3-22b-distilled-1.1.safetensors`; config repo `diffusers/LTX-2.3-Distilled-Diffusers`
- Also reproduced against `Lightricks/LTX-2` → `ltx-2-19b-dev.safetensors` (LTX 2.0) for bug 2

### Who can help?

@DN6 @a-r-r-o-w

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start in loaders/single_file_utils.py with convert_ltx2_transformer_to_diffusers, convert_ltx2_vae_to_diffusers, and convert_ltx2_audio_vae_to_diffusers, then trace LTX2Pipeline.from_single_file and load_single_file_sub_model. Run the supplied pipeline and model reproductions, checking key parity for the LTX 2.3 and 2.0 checkpoints. Done means all three components load their own weights and pipe.to("cuda") succeeds without meta tensors.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
backend, machine-learning
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.