huggingface / huggingface/diffusers

`LTX2*Pipeline.from_single_file` cannot load LTX 2.3 (or 2.0) combined checkpoints: unmapped 2.3 keys leave meta tensors, and the LTX2 converters drain the shared checkpoint dict

Offen
#14,145 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug models pipelines single-file
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

### Describe the bug

Pipeline-level `from_single_file` is broken for the Lightricks LTX-2 combined (flat) checkpoints. There are two independent bugs; both are present on current `main` (verified in source at commit `208704a`, 2026-07-08). **I intend to submit a pull request for this** — a minimal, verified fix is ready.

**Bug 1 — the LTX2 key converters don't map the 2.3-only key names, and unmapped keys fail only later, as meta tensors.**

`convert_ltx2_transformer_to_diffusers` (`loaders/single_file_utils.py`) does not map the 2.3 prompt-register modulation keys:

```
model.diffusion_model.prompt_adaln_single.* -> prompt_adaln.* (6 keys)
model.diffusion_model.audio_prompt_adaln_single.* -> audio_prompt_adaln.* (6 keys)
```

(`LTX2VideoTransformer3DModel` has had `prompt_adaln` / `audio_prompt_adaln` modules since #13217, but the converter was never extended.)

Likewise `convert_ltx2_vae_to_diffusers`' rename dict stops at the 2.0 decoder's `up_blocks.6`; the 2.3 video VAE decoder has one more up block:

```
vae.decoder.up_blocks.7.* -> decoder.up_blocks.3.upsamplers.0.* (2 keys)
vae.decoder.up_blocks.8.* -> decoder.up_blocks.3.* (16 keys)
```

Because single-file loading initializes the model under `init_empty_weights` and unconverted checkpoint keys are merely reported as "unexpected", the affected modules silently keep meta tensors. Nothing fails at load time — the error surfaces later, in `pipeline.to(device)`:

```
NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.
```

This bites even the model-level `LTX2VideoTransformer3DModel.from_single_file(...)`.

**Bug 2 — every LTX2 converter consumes the whole shared checkpoint dict, so only the first checkpoint-sourced component gets any weights.**

All three LTX2 converters (`convert_ltx2_transformer_to_diffusers`, `convert_ltx2_vae_to_diffusers`, `convert_ltx2_audio_vae_to_diffusers`) start with:

```python
converted_state_dict = {key: checkpoint.pop(key) for key in list(checkpoint.keys())}
```

In pipeline-level `from_single_file`, `load_single_file_sub_model` passes the **same** checkpoint dict to each component, and `FromOriginalModelMixin.from_single_file` calls `checkpoint_mapping_fn` on it without copying. The first component loaded (e.g. the transformer) therefore pops **all** keys — including `vae.*`, `audio_vae.*`, `vocoder.*` — and every later component receives an empty dict, failing with:

```
SingleFileComponentError: Failed to load AutoencoderKLLTX2Video. Weights for this component appear to be missing in the checkpoint.
```

This bug is version-independent: it affects LTX **2.0** combined checkpoints (e.g. `Lightricks/LTX-2` `ltx-2-19b-dev.safetensors`) the same way — as far as I can tell the pipeline-level single-file path for LTX2 has never worked (there are no LTX2 tests under `tests/single_file/`). The older `convert_ltx_vae_checkpoint_to_diffusers` (LTX-Video 0.9.x) already avoids this by slicing the dict by prefix (`... if "vae." in key`).

**Proposed fix** (contained to the three LTX2 converters in `loaders/single_file_utils.py`):

1. Add the 2.3 renames: transformer `prompt_adaln_single` → `prompt_adaln`, `audio_prompt_adaln_single` → `audio_prompt_adaln` (inert for 2.0, which lacks these keys); video VAE `up_blocks.7` → `up_blocks.3.upsamplers.0`, `up_blocks.8` → `up_blocks.3`.
2. Make each converter consume only its own prefix (`model.diffusion_model.` / `vae.` / `audio_vae.`), falling back to all keys when the prefix is absent (extracted single-component files) — mirroring `convert_ltx_vae_checkpoint_to_diffusers`.

Related follow-up (out of scope for the fix): `infer_diffusers_model_type` maps every LTX2 checkpoint to `ltx2-dev` → `Lightricks/LTX-2`, so a 2.3 checkpoint silently gets the 2.0 architecture config unless `config=` is passed explicitly. A `model.diffusion_model.prompt_adaln_single.*` key unambiguously identifies 2.3 and could drive a 2.3 default config.

cc @dg845 (author of #13217, which added LTX-2.3 support).

### Reproduction

```python
import torch
from diffusers import LTX2Pipeline

pipe = LTX2Pipeline.from_single_file(
"https://huggingface.co/Lightricks/LTX-2.3/blob/main/ltx-2.3-22b-distilled-1.1.safetensors",
config="diffusers/LTX-2.3-Distilled-Diffusers",
torch_dtype=torch.bfloat16,
)
pipe.to("cuda")
```

Depending on which component loads first you get either the `SingleFileComponentError` (bug 2) or, after working around bug 2, the meta-tensor `NotImplementedError` from `pipe.to()` (bug 1).

Model-level reproduction of bug 1 alone (no pipeline involved):

```python
import torch
from diffusers import LTX2VideoTransformer3DModel

transformer = LTX2VideoTransformer3DModel.from_single_file(
"https://huggingface.co/Lightricks/LTX-2.3/blob/main/ltx-2.3-22b-distilled-1.1.safetensors",
config="diffusers/LTX-2.3-Distilled-Diffusers",
subfolder="transformer",
torch_dtype=torch.bfloat16,
)
transformer.to("cuda") # NotImplementedError: Cannot copy out of meta tensor
# transformer.prompt_adaln.linear.weight.is_meta == True
```

**Key-level evidence (no GPU needed).** Running the three converters in sequence on one shared dict built from the real `ltx-2.3-22b-distilled-1.1.safetensors` header (dummy values; the converters only move keys), and diffing each result against the matching component of `diffusers/LTX-2.3-Distilled-Diffusers`:

Current `main`:

```
[transformer] converted=5689 expected=4186 missing=12 unexpected=1515
MISSING prompt_adaln.* / audio_prompt_adaln.* (12 keys)
UNEXPECTED *_single (unmapped) + vae.* + audio_vae.* + vocoder.* (swallowed from the other components)
[vae] converted=0 expected=170 missing=170 # shared dict already drained by the transformer converter
[audio_vae] converted=0 expected=102 missing=102 # shared dict already drained
```

`convert_ltx2_vae_to_diffusers` in isolation (only `vae.*` keys), showing the up-block gap:

```
missing: 18 unexpected: 18
UNEXPECTED decoder.up_blocks.7.*, decoder.up_blocks.8.*
MISSING decoder.up_blocks.3.upsamplers.0.*, decoder.up_blocks.3.resnets.*
```

With the proposed fix, all three components reach exact key parity for **both** generations:

```
LTX 2.3 (ltx-2.3-22b-distilled-1.1 vs diffusers/LTX-2.3-Distilled-Diffusers):
[transformer] 4186/4186 [vae] 170/170 [audio_vae] 102/102 (0 missing, 0 unexpected)
LTX 2.0 regression check (ltx-2-19b-dev vs Lightricks/LTX-2):
[transformer] 3510/3510 [vae] 184/184 [audio_vae] 102/102 (0 missing, 0 unexpected)
```

After conversion, only `vocoder.*` and `text_embedding_projection.*` remain in the shared dict — expected, as those are not single-file-loadable and come from the config repo.

### Logs

```shell
Meta-tensor failure from `pipe.to()` / `transformer.to()` (bug 1):

NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

Starved-component failure (bug 2), for whichever checkpoint-sourced component loads after the first:

SingleFileComponentError: Failed to load AutoencoderKLLTX2Video. Weights for this component appear to be missing in the checkpoint.
```

### System Info

- `diffusers` version: 0.39.0.dev0 (git main). The bug is also confirmed by reading the source on `main` at commit `208704a` (2026-07-08).
- Platform: Linux aarch64 (NVIDIA DGX Spark / GB10)
- Python: 3.12
- PyTorch: 2.x (cu130)
- `transformers`, `safetensors`, `huggingface_hub`: current releases
- Checkpoint: `Lightricks/LTX-2.3` → `ltx-2.3-22b-distilled-1.1.safetensors`; config repo `diffusers/LTX-2.3-Distilled-Diffusers`
- Also reproduced against `Lightricks/LTX-2` → `ltx-2-19b-dev.safetensors` (LTX 2.0) for bug 2

### Who can help?

@DN6 @a-r-r-o-w

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne in loaders/single_file_utils.py mit convert_ltx2_transformer_to_diffusers, convert_ltx2_vae_to_diffusers und convert_ltx2_audio_vae_to_diffusers und verfolge anschließend LTX2Pipeline.from_single_file und load_single_file_sub_model. Führe die bereitgestellten Pipeline- und Modell-Reproduktionen aus und überprüfe die Schlüsselparität für die LTX 2.3- und 2.0-Checkpoints. Die Aufgabe ist abgeschlossen, wenn alle drei Komponenten ihre eigenen Gewichte laden und pipe.to("cuda") ohne Meta-Tensoren erfolgreich ist.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
backend, machine-learning
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.