Token Merging doubles vram usage

When using the `TomePatchModule` with a ratio of at least 0.01, the VRAM usage doubles. This increase is most noticeable when using high resolutions. In my case, VRAM usage is not doubled for resolutions below approximately 1260x1260. However, I suspect that even at lower resolutions, it still doubles but it's just barely noticeable. To reproduce this issue clearly, I recommend using a resolution of 2048x2048.

The red box in the graph represents VRAM usage when using Token Merging, while the blue box represents VRAM usage without it. I have also provided the workflow I followed for reference.

[workflow.txt](https://github.com/comfyanonymous/ComfyUI/files/12751179/workflow.txt)