GPU crashes regularly while using API (and other times)

There are times when I can run ComfyUI manually for hours, doing lots of different things, swapping out LoRAs, etc. but there are certain situations that consistently crash the GPU. And I don't mean the computer or the video device driver, I mean the video card stop responding on the bus. The computer no longer detects the video card at all. Attempting to install the device driver tells me there is no video card in the system. A reboot is required to get the video card to respond again.

The most consistent way to get it to happen is for me to run a fairly simple prompt over and over using the API (I'm changing the prompt with every run of four images). It crashes pretty consistently every 100 images generated. If I kill and restart the ComfyUI server every 90 images, then it crashes about every 200 images. I'm trying some different things, like pausing between restarts to see if it behaves the same, but it takes a while to run that way.

I don't know how to take ComfyUI out of the loop to see if it's really just Stable Diffusion that is causing the problem, so I figured I come here to ask if there's something I could be doing to isolate the issue (or to stop having the issue at all). Also, it seems that this more common failure rate happened after I did the last pull of ComfyUI, but frankly I'm still learning the system so I've been doing a lot of very different things. This is my first project where I'm generating several thousand images driven by a program.

Here is my configuration:
Processor: 12th Gen Intel(R) Core(TM) i9-12900K, 3187 Mhz, 16 Core(s), 24 Logical Processor(s)
Motherboard: Z690 Taichi
Installed Physical Memory (RAM) 32.0 GB (DDR5)
NVIDIA GeForce RTX 3090 24GB VRAM
Video driver 31.0.15.4592 (2023-10-19)
DirectX 12

Here are the things that I've essentially ruled out:
This is not a temperature issue, I'm fully water cooled and the video card hovers right around 62C (140F), or about 70F above ambient, it never thermal throttles. It's also not a video memory issue since it loads up 10.5GB while it's running the iterations and then momentarily jumps up to 17.3GB as it generates the final image files, still showing plenty of headroom (and is extremely consistent since the changes I'm making to the prompt are small). The GPU never gets above 10% utilization. The normal memory never gets above 17GB. CPU usage hovers right around 30%. I'm not doing any overclocking of the CPU or the GPU during these runs. Also, I've never had this problem while playing high-end games or running heavy GPU loads like transcoding.