huggingface / huggingface/diffusers

[PAG] - Adaptive Scale bug

Open
#9,930 1 comment 1 reaction 0 assignees View on GitHub
bug stale
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Describe the bug

I am looking for the purpose of the PAG adaptive scale? Because I was passing a value in it, for example 5.0, and passing 3.0 in the PAG scale, according to the implemented code we will have a negative number and the scale will return 0 and the PAG will not be applied and I did not find an explanation about this parameter in the documentation.

So i found it on an ComfyUI documentation: "_This dampening factor reduces the effect of PAG during the later stages of the denoising process, speeding up the overall sampling. A value of 0.0 means no penalty, while 1.0 completely removes PAG_"

Then I realized that I was passing values ​​above 1.0, however when I pass values ​​of 0.2 it is enough for it not to apply the PAG. I suspect this could be a problem.

If you run the code below, you will see that in the third image where I pass a scale of 0.2 in adaptive_scale it practically invalidates the PAG in the first generation steps.

I propose a possible solution:

After this code:
https://github.com/huggingface/diffusers/blob/5c94937dc7561767892d711e199f874dc35df041/src/diffusers/pipelines/pag/pag_utils.py#L93

We can change for:
```python
if self.do_pag_adaptive_scaling:
signal_scale = self.pag_scale
if t / self.num_timesteps > self.pag_adaptive_scale:
signal_scale = 0
return signal_scale
else:
return self.pag_scale
```
And inside every PAG pipeline, we need change "t" variable for "i" variable is passed with param on this function, to receive the number of current step.

https://github.com/huggingface/diffusers/blob/5c94937dc7561767892d711e199f874dc35df041/src/diffusers/pipelines/pag/pipeline_pag_sd_xl.py#L1253

With this, the logic will not be that the higher the adaptive scale value, the faster the PAG will be disabled, but quite the opposite. The scale will tell you exactly at what point in the process the PAG will be disabled. If the scale exceeds 0.5 in a 30-step generation, the PAG will be disabled from step 15 onwards. The scale applied will be the same until the moment of the cut and will not be a variable scale.
I don't know if this was the original purpose of this parameter, but it works well for me.

### Reproduction

```python
from diffusers import AutoPipelineForText2Image
import torch

device = "cuda"

pipeline_sdxl = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
enable_pag=True,
pag_applied_layers=["mid"],
torch_dtype=torch.float16
).to(device)

pipeline = AutoPipelineForText2Image.from_pipe(pipeline_sdxl, enable_pag=True).to(device)
pipeline.enable_vae_tiling()
pipeline.enable_model_cpu_offload()

prompt = "an insect robot preparing a delicious meal, anime style"

for i, pag_scale in enumerate([0.0, 3.0, 3.0]):
generator = torch.Generator(device="cpu").manual_seed(0)
images = pipeline(
prompt=prompt,
num_inference_steps=25,
guidance_scale=7.0,
generator=generator,
pag_scale=pag_scale,
pag_adaptive_scale=0.0 if i < 2 else 0.2
).images[0]
images.save(f"./data/result_pag_{i+1}.png")

```

### Logs

```shell
N/A
```

### System Info

- 🤗 Diffusers version: 0.32.0.dev0
- Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Running on Google Colab?: No
- Python version: 3.10.11
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Flax version (CPU?/GPU?/TPU?): 0.10.1 (cpu)
- Jax version: 0.4.35
- JaxLib version: 0.4.35
- Huggingface_hub version: 0.26.2
- Transformers version: 4.46.2
- Accelerate version: 1.1.1
- PEFT version: 0.13.2
- Bitsandbytes version: not installed
- Safetensors version: 0.4.5
- xFormers version: 0.0.27.post2
- Accelerator: NVIDIA GeForce RTX 3060 Ti, 8192 MiB
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help?

@yiyixuxu , @asomoza

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.