Comfy-Org / Comfy-Org/ComfyUI

BF16 WeightHooks can produce black output after model offload

Open
#15,531 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

## Problem

A mapped model `WeightHook` can produce an all-black image on a BF16 Krea2
checkpoint even though the prompt completes successfully and the same LoRA
works through `LoraLoaderModelOnly`.

This was found while following up #14382, but it is independent of the
quantized state-key fix in #14413 and the setter/cache issue in #15530.

## Reproduction

With DynamicVRAM enabled, compare the same public LoRA and seed:

1. No LoRA: normal image.
2. `LoraLoaderModelOnly`: normal LoRA result.
3. `CreateHookLoraModelOnly -> SetClipHooks`: successful request but
black/non-finite output.

A mapped zero-valued patch also fails, while an unmatched zero-patch Hook is
normal. Patch calculation and the first layer output remain finite. Intercepting
the final write or disabling pinned asynchronous offload avoids the failure.

## Expected behavior

Hook writeback after model preparation/offload must preserve a valid storage
and transfer lifecycle. A mapped zero-valued Hook should match the no-LoRA
control, and a non-zero Hook should remain finite and nonblack.

## Validation

The local candidate keeps plain-weight Hook transfers synchronous and restores
declared model dtype casting. A mapped BF16 subset then completes two
consecutive prompts without the black image.

The full public BF16 Hook exceeds the available 30 GiB host-memory envelope on
the current test system and is not claimed by this report.

Contributor guide

Open the contributing guide

Research direction

Start with the mapped model WeightHook path exercised by CreateHookLoraModelOnly -> SetClipHooks, then reproduce with DynamicVRAM while comparing plain-weight transfers and pinned asynchronous offload. Done means mapped BF16 zero-valued Hooks match the no-LoRA control and non-zero Hooks remain finite and nonblack across two consecutive prompts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.