LTX-2.3 pipeline: native SIGSEGVs on tpu7x

> [!IMPORTANT]
> **Update (2026-06-22): the 1024x1536 denoise compile is itself unreliable; the SIGSEGV looks host/run-dependent, not LoRA- or perturbation-specific.**
> Controlled re-testing changes the picture below. The "1024x1536 known-good" baseline referenced throughout is **not reliable**: the identical config (no LoRA, compile-cache off) produced a clip on one VM, but **segfaulted at the denoise compile on a second VM on every attempt (5/5)** — a controlled repeat confirmed it's not run-to-run flaky on that host. Implications:
> - **2b is not a distinct trigger** — the same crash happens with **no LoRA** at the same config. The LoRA fuse is clean (0 unmatched keys); generating with it just hits the same base crash.
> - The "1024 works, a perturbation tips it over" framing (2a, 2e, and the Unifying observation) is **suspect**, because the 1024 baseline it's measured against is itself flaky.
> - **v6e (4-chip) ran the same LTX-2.3 model e2e cleanly** (`GEN_EXIT=0`) at 512x768x121 — both no-LoRA and with a colorization LoRA fused (video produced, profiling pass and all). So the failure is specific to the v7 host(s) we hit, not the model/config/pipeline.
> - Working hypothesis: **host/instance-dependent libtpu/XLA instability** in the denoise compile, not tied to a config knob. Same jax/libtpu versions and command across working and failing runs; the difference is the physical VM. (Evidence, not proof — few hosts sampled, and the v6e run differs in mesh/resolution.) Investigation ongoing.

## Summary

Running the LTX-2.3 path (`configs/ltx2_3_video.yml` + the default `dg845/LTX-2.3-Diffusers` weights) on a single tpu7x (Ironwood, 1 chip = 2 cores, 192 GB HBM) hits several native segmentation faults (`Fatal Python error: Segmentation fault`, process exit 139).
The LTX-2 path (`configs/ltx2_video.yml` + `Lightricks/LTX-2`) on the same box is stable, so these are 2.3-specific.

They fall into two buckets:

- **(1) Bad-config cases that should raise a clear error instead of segfaulting.** A user passing an unsupported/invalid setting gets an opaque native crash with no actionable message.
- **(2) Opaque compiler crashes with no obvious bad input: these need investigation.** Including the repo's own default config resolution.

Environment:
Ubuntu 24.04, Python 3.12, `jax==0.9.0`, `jaxlib==0.9.0`, `libtpu==0.0.34` (stable `jax[tpu]`), 1x `tpu7x-standard-1t`. Single host (`skip_jax_distributed_system=True`).

A reference config (the bare path everything below deviates from):
**`ltx2_3_video.yml` at 1024x1536x121, 40 steps, `a2v_attention_kernel=dot_product`, `per_device_batch_size=0.5 ici_data_parallelism=1 ici_context_parallelism=2`, no LoRA, compile-cache OFF**. This generated a video in ~255 s warm on one VM, but per the Update above it segfaults at the denoise compile on another VM, so treat it as a *reference command*, not a reliable baseline.

---

## Bucket 1: should error clearly, currently segfaults

### 1a. `per_device_batch_size * num_devices` not a positive integer → SIGSEGV at warmup
The 2.3 config default is `per_device_batch_size: 1.0` and `ici_context_parallelism: -1`. On a 2-core host with no explicit sharding, the warmup compile segfaults instead of validating that `per_device_batch_size * num_devices` is a shardable positive integer.

Repro:
```bash
python src/maxdiffusion/generate_ltx2.py src/maxdiffusion/configs/ltx2_3_video.yml \
skip_jax_distributed_system=True a2v_attention_kernel=dot_product \
run_name=repro_1a # uses config-default batch/sharding on a 2-core host
```
Expected: a clear error ("per_device_batch_size (1.0) * num_devices (2) = 2 must be ... / mesh axes must multiply to num_devices"). Actual: `Fatal Python error: Segmentation fault` (exit 139).

### 1b. Two-stage latent upsampler with mismatched config/weight provenance → SIGSEGV
`run_latent_upsampler=True` builds the upsampler with the **config** from `Lightricks/LTX-2` (`pipelines/ltx2/ltx2_pipeline.py` `load_upsampler`, ~L595-660) but **weights** from raw `Lightricks/LTX-2.3`: config dims vs weight shapes mismatch → segfault during upsampler construction (before any denoise), after `Upsampler config inferred: ... scale=None`.

Expected: a shape/provenance mismatch error. Actual: SIGSEGV. (A consistent-provenance source like `dg845/LTX-2.3-Spatial-Upsampler-Diffusers` fixes the config inference, but then a *separate* crash appears, see 2c.)

---

## Bucket 2: opaque compiler crashes, need investigation

All of these crash with the same signature: `Fatal Python error: Segmentation fault` deep in `jax/_src/compiler.py backend_compile_and_load` (the libtpu/XLA compile), no Python-level cause, always at the same point (right after "Connectors pass"). **NB (see Update): the bare 1024x1536 denoise compile is itself unreliable** - it succeeded on one VM but segfaults on another - so the "a perturbation tips a working baseline over" reading is suspect; these may all be the same host/run-dependent crash observed under different conditions. Cases kept below for their specific repros.

### 2a. The repo's DEFAULT 2.3 resolution (512x768) segfaults at warmup [no LoRA, no cache]
The most basic run: `ltx2_3_video.yml` ships `height: 512`, `width: 768`, and a clean run at that default resolution segfaults at the warmup denoise compile (right after "Connectors pass"), with the compile cache **off** and **no** LoRA. 1024x1536 *sometimes* works (but per the Update it's unreliable). If 512x768 turns out to fail more consistently than 1024 that would point to a shape dependence on top of the host/run instability - but that needs the controlled comparison; for now treat 2a as "the default resolution also segfaults," not a proven shape bug.

Repro (the config default + only the required single-host/sharding/audio-kernel flags):
```bash
python src/maxdiffusion/generate_ltx2.py src/maxdiffusion/configs/ltx2_3_video.yml \
skip_jax_distributed_system=True a2v_attention_kernel=dot_product \
per_device_batch_size=0.5 ici_data_parallelism=1 ici_context_parallelism=2 \
height=512 width=768 run_name=repro_2a
```

### 2b. (correction) Fusing a LoRA is NOT a distinct trigger
This earlier looked LoRA-specific; it isn't. The LoRA **fuses cleanly** (0 unmatched keys), but generating with it segfaults at the denoise compile - and the **identical config with no LoRA also segfaults** (the bare-1024 crash noted in the Update). So the LoRA *loading* is fine; *generating* just hits the same host/run-dependent base crash, not a LoRA-specific one. Repro: run the reference 1024x1536 command with and without `enable_lora=True` and compare.

### 2c. Profiling-pass recompile segfaults AFTER a successful generation
Hits every successful run: the clip is generated and saved, then the built-in 5-step "profiling pass" recompiles the diffusion loop and segfaults (exit 139). Non-blocking for output (the video is already saved) but it crashes the process with a scary trace. Repro: any successful 1024x1536 run.

### 2d. Enabling the JAX persistent compile cache segfaults the 2.3 compile
An opt-in optimization: with `JAX_COMPILATION_CACHE_DIR` set, the 1024x1536 run (which succeeds with the cache OFF) segfaults at the warmup compile, in `_compile_and_write_cache -> backend_compile_and_load`. The same cache works for the LTX-2 path. (Possibly a jax/libtpu cache-serialization interaction with the 2.3 graph rather than a maxdiffusion bug - flagging for triage.) Repro: known-good 1024x1536 command + `JAX_COMPILATION_CACHE_DIR=/some/dir`.

### 2e. Latent upsampler loaded onto the mesh destabilizes the denoise → SIGSEGV
The advanced two-stage path: with consistent-provenance upsampler weights (so 1b's config bug is gone), `run_latent_upsampler=True` segfaults at the **denoise** compile (right after "Connectors pass"). Confirmed at BOTH 512x768 and **1024x1536** (the resolution that denoises fine bare) - so merely loading the upsampler onto the mesh destabilizes the otherwise-working denoise compile, it is not the 2a shape bug. Repro: known-good 1024x1536 command + `run_latent_upsampler=True upsampler_model_path=dg845/LTX-2.3-Spatial-Upsampler-Diffusers upsampler_filename=diffusion_pytorch_model.safetensors`.