ai-forever / ai-forever/Kandinsky-2

How to use the ddim_reverse_sample method?

オープン
#67 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Jupyter Notebook
スター
2.8k
フォーク
319
PR マージ指標
30日以内にマージされた PR はありません

説明

Hi, I have a question about using the DDIM inversion scheduler with Kandinsky 2_1.

Can it be directly operated with [this method](https://github.com/ai-forever/Kandinsky-2/blob/47c01fa310ce036da46b6335d9f99e66edfdaab2/kandinsky2/model/gaussian_diffusion.py#L521)?

```python
...
@torch.no_grad()
def test(
self,
num_steps = 100,
batch_size = 1,
guidance_scale = 4,
h = 768,
w = 768,
sampler = 'ddim_sampler',
prior_cf_scale = 4,
prior_steps = "5" ,
negative_prior_prompt = "",
negative_decoder_prompt = ""):

config = deepcopy(self.config)
diffusion = create_gaussian_diffusion(**config["diffusion_config"])

init_image = Image.open('images/test.png')
image = prepare_image(pil_image=init_image, h=h, w=w).to(self.device)
if self.use_fp16:
image = image.half()
image = self.image_encoder.encode(image) * self.scale
print(image.shape, image.dtype)

t = torch.tensor([diffusion.timestep_map[999]]).to(self.device)
print(t, t.shape)

image_noise = diffusion.ddim_reverse_sample(model=self.model, x=image, t=t)
...

```
```
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
torch.Size([1, 4, 96, 96]) torch.float16
tensor([999], device='cuda:0') torch.Size([1])
Traceback (most recent call last):
File "/home/kandinsky2/code/bdAi/test.py", line 282, in
model.test()
File "/home/kandinsky2/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/kandinsky2/code/bdAi/test.py", line 260, in test
image_noise = diffusion.ddim_reverse_sample(model=self.model, x=image, t=t)
File "/home/kandinsky2/lib/python3.9/site-packages/kandinsky2/model/gaussian_diffusion.py", line 535, in ddim_reverse_sample
out = self.p_mean_variance(
File "/home/kandinsky2/lib/python3.9/site-packages/kandinsky2/model/respace.py", line 102, in p_mean_variance
return super().p_mean_variance(self._wrap_model(model), *args, **kwargs)
File "/home/kandinsky2/lib/python3.9/site-packages/kandinsky2/model/gaussian_diffusion.py", line 251, in p_mean_variance
model_output = model(x, s_t, **model_kwargs)
File "/home/kandinsky2/lib/python3.9/site-packages/kandinsky2/model/respace.py", line 133, in __call__
return self.model(x, new_ts, **kwargs)
File "/home/kandinsky2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kandinsky2/lib/python3.9/site-packages/kandinsky2/model/text2im_model2_1.py", line 88, in forward
text_outputs = self.get_text_emb(
File "/home/kandinsky2/lib/python3.9/site-packages/kandinsky2/model/text2im_model2_1.py", line 61, in get_text_emb
clip_seq = self.clip_to_seq(image_emb).reshape(
File "/home/kandinsky2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kandinsky2/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
TypeError: linear(): argument 'input' (position 1) must be Tensor, not NoneType
```

In a similar way to the method [generate_img2img](https://github.com/ai-forever/Kandinsky-2/blob/47c01fa310ce036da46b6335d9f99e66edfdaab2/kandinsky2/kandinsky2_1_model.py#L428). For example, if I want to have the inverted latent for the image embedding at the output of the encoder.

I am not sure of my understanding. Here the argument model in `diffusion.ddim_reverse_sample`, can it be directly `self.model?`

And for the argument` t`, if we want the same number of steps, so 100?

After that, I want to pass as an argument to the [generate_img(](https://github.com/ai-forever/Kandinsky-2/blob/47c01fa310ce036da46b6335d9f99e66edfdaab2/kandinsky2/kandinsky2_1_model.py#L472) method as `noise`.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。