Color fringe artifacts on whole image when inpainting->save->load->inpaint->etc multiple times in a row

Even though I'm using masking, the whole image will progressively get more blown out with color fringing/chromatic aberration looking artifacts.

Here's the basic gist of my workflow:
Make something
Save it to png
Load it into an image node
Mask something off
Sample to change masked area until I get what I want
Save it again
Load that new save into an image node
Mask off new regions
etc etc

In terms of what I'm using, I'm just jusing the base SDXL 1.0 base, sometimes the SDXL 1.0 refiner(usually not) and the SDXL 1.0 vae. I've tested the issue with regular masking->vae encode->set latent noise mask->sample and I've also tested it with the load unet SDXL inpainting 0.1 model->mask->vae encode for inpainting-sample. In terms of samplers, I'm just using dpm++ 2m karras and usually around 25-32 samples, but that shouldn't be causing the rest of the unmasked image to be touched anyways.

After a few or so round-trips of that, the rest of the untouched image will progressively get worse and worse. I'm thinking that maybe it's a floating point rounding issue that keeps rounding upward? Or maybe it has something to do with the vae encode->vae decode portion like a precision/rounding issue in there? I can partially mitigate it by using something like Ultimate SD Upscale with a low denoise value like 0.25 or 0.3, but I'd rather not have to do that every single loop through.

Any ideas?