LoRA-patched FP8 weights are not run-to-run deterministic under dynamic VRAM (comfy_aimdo)

# LoRA-patched FP8 weights are not run-to-run deterministic under dynamic VRAM (comfy_aimdo)

**Environment**
- ComfyUI with `comfy_aimdo` 0.4.10 (dynamic VRAM / vbar paging is the default via `main.py:278` → `ModelPatcherDynamic`)
- RTX 4070 12 GB, driver 610.43.03
- Model: Krea2 UNET in fp8 (`float8_e4m3fn`, straight-cast, runtime scale = 1.0, ~6.5 GB of weights)
- LoRA: `Krea2-realism-V2.safetensors` — entirely LoKr (256 modules of `lokr_w1` [4,4] + `lokr_w2` + `alpha`), applied at strength 1.0
- Sampler: `er_sde`, 8 steps, **fixed seed across all runs**

## Summary

With the **same seed and the same graph**, running the workflow twice produces slightly different images (~5–7/255 mean abs diff, p99.9 ≈ 100–110) **only when a LoRA is loaded**. Removing the LoRA makes the identical two runs **pixel-identical** (mean diff 0.0, max 0).

This breaks ComfyUI's reproducibility guarantee (same seed ⇒ same image) specifically for the fp8 + LoRA + dynamic-VRAM combination. All isolated components are deterministic; the non-determinism lives in the interaction between the dynamic-VRAM (vbar) paging path and LoRA-patched FP8 weights.

## Reproduction

1. Load an fp8-scaled diffusion model larger than VRAM (dynamic VRAM active).
2. Load any LoRA at strength 1.0 (LoKr or standard LoRA; LoKr reproduces it).
3. Queue the **same prompt twice** with the **same seed**.
4. Diff the two outputs — they differ slightly.
5. Remove the LoRA node, queue twice again — the outputs are bit-identical.

## Expected

Same seed + same graph ⇒ identical output (holds for no-LoRA runs).

## Actual

With a LoRA loaded, outputs drift run-to-run (~6/255). Verified across many runs, including a same-loader pair and a value-neutral graph variant (extra `ConditioningZeroOut` node) — no-LoRA stays bit-identical regardless of the graph variant; any run with the LoRA differs from any other by the same noise magnitude.

## What was ruled out

All isolated with fixed inputs, each bit-identical across processes:
- Noise sampler (`er_sde` → `default_noise_sampler(x, seed)` — seed reaches the sampler via `extra_args["seed"]`, `samplers.py:1228`)
- FP8 GEMM (`_scaled_mm` / comfy_kitchen fp8 linear)
- SDPA attention (48 heads × 4096 tokens × 128 dim)
- comfy_kitchen quantize ops / `requantize_from_float` (round-to-nearest, ignores `stochastic_rounding`)
- LoKr patch math (SHA-identical across 3 processes)
- Full per-forward patched layer: fp8 dequant → LoKr patch → fp16 `F.linear` (SHA-identical across processes)
- Loader equivalence: native `load_lora_for_models` vs a block-weighted loader produce the **identical** 256-adapter set (same keys/types/shapes, zero dropped) — the drift is not loader-related

## Likely mechanism (analysis, not yet observed inside the compiled lib)

For LoRA-patched keys with matching shapes, `model_patcher.py:1901-1903` installs a `LowVramPatch` as the module's `weight_lowvram_function`, and the module still receives a vbar allocation (`m._v = vbar.alloc(...)`, `model_patcher.py:1956`). On **every page-in** of a patched layer, the dynamic-VRAM path (`comfy/ops.py`, `resolve_cast_module_with_vbar`) does:

1. dequant fp8 → fp16 (`to_dequant`)
2. apply the LoRA diff (`lowvram_fn(x)`, `LowVramPatch.__call__` → `calculate_weight`)
3. **stochastic-round back to fp8** (`comfy.float.stochastic_rounding`, seed = `string_to_seed(seed_key)`, ops.py ~306-311)
4. **bake in-place** into the weight (`orig.copy_(y)`, ops.py:315)

The rounding is deterministically seeded, so a single bake is reproducible — but the **page-in / evict / re-fault schedule is the only part of the pipeline not fixed by the seed**, and the bake is not idempotent when a patch is present (a re-faulted layer can be patched again on top of a previously baked value, and the effective per-layer weight then depends on how many times the page was faulted). No-LoRA is immune because requantizing an already-quantized fp8 value is idempotent (fp8→fp32→fp8 is exact), so paging noise cancels.

A secondary observation consistent with this: a one-time deterministic merge of the same LoRA into the model (exact strength 1.0, same stochastic-rounding seeds) reproduces the runtime's overall effect magnitude exactly but still differs from any single runtime run by ~11.7/255 — more than the ~6/255 run-to-run noise — suggesting the runtime's per-layer effective strengths are not exactly 1.0 everywhere.

## Additional finding: RNE requant silently halves the LoRA effect

While testing the above, we found that **round-to-nearest** fp8 requantization of LoRA-patched weights keeps only **~30–62% (avg 47%)** of the LoRA delta — fp8 E4M3's ~12.5% relative step swallows sub-ULP deltas. The runtime's stochastic rounding is what preserves the full effect (100% in expectation, seeded → deterministic). This is working as designed in the runtime, but it means any requant path that drops to RNE (e.g., a merge tool, or a save/load cycle) silently weakens LoRAs by ~half on fp8 models. Worth documenting / guarding.

## Why this is fixable without changing the memory-management trade-off

To preempt the "dynamic VRAM is best-effort, non-determinism is the intended trade" dismissal:

- The **same** dynamic-VRAM machinery is bit-reproducible when no LoRA is loaded (requantizing an already-quantized fp8 value is idempotent). The trade is not "determinism vs memory" — it is "determinism vs LoRA", which no design intends.
- The suggested fixes do not alter memory management or add patching cost: per-page-in work (dequant → patch → requant → bake) stays identical in fix (a), and is reduced in fix (b). Only the force-load option trades VRAM, and it is optional.
- Measured on the repro machine: no-LoRA generations 24–28 s, LoRA generations 28–36 s — the patched-forward path is heavier, but the fixes neither add to nor depend on that delta.

## Suggested fix direction

- Make the vbar page-in bake idempotent for patched weights: either reload the **original** weight from the backing file on re-fault (so each fault = original + 1× diff), or skip re-patching when the page was already baked in this run (track bake state per layer), so the effective strength is exactly `strength` regardless of paging.
- Alternatively, keep patched layers out of the vbar paging path (force-load them once), accepting the VRAM cost.

## Artifacts

- 8 output images from the A/B runs, including the pixel-identical no-LoRA pair, and PNG-embedded prompts for exact graph/seed verification (available on request).
- Isolation harnesses for every deterministic component (available on request).

Reproduce the fixed-seed LoRA/no-LoRA comparison first, then trace main.py:278 into ModelPatcherDynamic and the vbar path in model_patcher.py:1901-1903, 1956 and comfy/ops.py:306-315. Confirm that patched layers are not accumulated across page-ins and that repeated LoRA runs become pixel-identical while no-LoRA behavior remains unchanged; use the reported isolation harnesses if available.

Written by the indexing model from the issue text.