question about the momentum encoder

Hi. Thanks for your work. This is a question related to the paper, not the code. It may be a stupid question but I would love to hear an explanation from you.

Your main reason for using a momentum encoder is to achieve consistency, in the sense that if you don't update the key encoder with a slow moving average of the query encoder and simply use a copy of the query encoder, the features of previous mini-batches you stored in the queue become inconsistent due to the rapidly changing query encoder when you update it. So the momentum encoder basically makes the inconsistency among all these different mini-batch features small.

My question is why can't you simply make the stored keys as the input image views (before running them through the key encoder) rather than the features of these image views, and then for every update (when you need the negative samples) you just run these stored image views (negative samples) through a copy of the same query encoder? All of the output features you get would be consistent as well.