SDXL crashes after exactly 2 generation.

I have RTX 3060 12 GB and 16 GB RAM, i5 processor. Stable diffusion 1.5 works perfectly.
I was able to load SDXL base successfully. At first it crashed every time I "queue prompt". However after closing unnecessary application I was able to make it work. Now it works fine for first 2 generation and crashes when I try 3rd generation. It is weird because why crash after exactly 2 generation? I am loading SDXL base model and adding to queue.

I tried using pytorch-cross-attention, and dont-upcast-attention but they too crash after exactly two generation.

First generation take 180 or so seconds but second generation take only 20 or so seconds.

```
Setting up MemoryEfficientCrossAttention. Query dim is 640, context_dim is 2048 and using 10 heads.
model_type EPS
adm 2816
making attention of type 'vanilla-xformers' with 512 in_channels
building MemoryEfficientAttnBlock with 512 in_channels...
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla-xformers' with 512 in_channels
building MemoryEfficientAttnBlock with 512 in_channels...
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
torch.Size([1, 1280]) 1024 1024 0 0 1024 1024
torch.Size([1, 1280]) 1024 1024 0 0 1024 1024
100%|███████████████████████████████████████████| 20/20 [00:20<00:00, 1.00s/it]
Prompt executed in 155.30 seconds
got prompt
torch.Size([1, 1280]) 1024 1024 0 0 1024 1024
torch.Size([1, 1280]) 1024 1024 0 0 1024 1024
100%|███████████████████████████████████████████| 20/20 [00:13<00:00, 1.50it/s]
Prompt executed in 21.74 seconds
got prompt
Killed
```
Apparently it is out of memory problem.

```Out of memory: Killed process 13718 (python3) total-vm:51390488kB, anon-rss:14263076kB, file-rss:78232kB, shmem-rss:4108kB, UID:1000 pgtables:35332kB oom_score_adj:0```

Are there any solution?