Plugging vector-quantize-pytorch into taming-transformers

Hi,

I noticed your architecture could be plugged within the pipeline from https://github.com/CompVis/taming-transformers. I have proposed a code here (https://github.com/tanouch/taming-transformers) doing that. It enables to properly compare the different features proposed in your repo (Lower codebook dimension, Cosine similarity, Orthogonal regularization loss, etc) with the original formulation.

The code from this repo can be seen in both files
- taming-transformers/taming/models/vqgan.py
- taming-transformers/taming/modules/vqvae/quantize.py

As you can see, it is easy to launch a large scale training with your proposed architecture.

I am not sure this issue belongs here or in the taming-transformers repo. However, I thought you might be interested.
Thanks again for your work and these open-sourced repositeries !

Start by reviewing the proposed code in the linked taming-transformers repository, especially taming-transformers/taming/models/vqgan.py and taming-transformers/taming/modules/vqvae/quantize.py. Compare how vector-quantize-pytorch is connected to the taming-transformers pipeline and determine whether this repository or taming-transformers should own the integration. Done would mean an agreed, maintained integration path.

Written by the indexing model from the issue text.