Learning codebooks also updates the encoder ?

Nobody has claimed this yet.

In the case of a learnable codebook, it seems to me that the encoder outputs are not completely detached before they are used to compute the [codebook loss](https://github.com/lucidrains/vector-quantize-pytorch/blob/master/vector_quantize_pytorch/vector_quantize_pytorch.py#L1138) because they are still connected to the loss indirectly via [distance matrix](https://github.com/lucidrains/vector-quantize-pytorch/blob/master/vector_quantize_pytorch/vector_quantize_pytorch.py#L695) and [quantized vectors ](https://github.com/lucidrains/vector-quantize-pytorch/blob/master/vector_quantize_pytorch/vector_quantize_pytorch.py#L718). The encoder still accumulates gradients (even though they are not updated by the in-place optimizer). Should not the encoder outputs be detached also before they are used to compute the distances ?

Start in vector_quantize_pytorch/vector_quantize_pytorch.py at the codebook loss around line 1138, then trace the distance matrix around line 695 and quantized vectors around line 718. Verify whether encoder outputs remain connected to the codebook-loss computation through those paths. Done means confirming the intended detachment behavior and adding or updating coverage for encoder gradients if the repository's existing tests provide a suitable entry point.

Written by the indexing model from the issue text.