flux_minimal_inference: CUDA Out of Memory with LORA and negative_prompt

I have 24Gb VRAM. flux_minimal_inference run without any problem (with or without negative_prompt). Using trained LORA with flux_minimal_inference (without negative_prompt) also do not causing any problems.

But using flux_minimal_inference with **LORA** and **negative_prompt** **together** causing **CUDA Out of Memory**

```
Traceback (most recent call last):
File "/opt/flux/sd-scripts/flux_minimal_inference.py", line 509, in
generate_image(
File "/opt/flux/sd-scripts/flux_minimal_inference.py", line 322, in generate_image
x = do_sample(
^^^^^^^^^^
File "/opt/flux/sd-scripts/flux_minimal_inference.py", line 174, in do_sample
x = denoise(
^^^^^^^^
File "/opt/flux/sd-scripts/flux_minimal_inference.py", line 111, in denoise
pred = model(
^^^^^^
File "/opt/conda/envs/flux/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/flux/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/flux/sd-scripts/library/flux_models.py", line 1050, in forward
img = block(img, vec=vec, pe=pe, txt_attention_mask=txt_attention_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/flux/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/flux/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/flux/sd-scripts/library/flux_models.py", line 858, in forward
return self._forward(x, vec, pe, txt_attention_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/flux/sd-scripts/library/flux_models.py", line 833, in _forward
attn = attention(q, k, v, pe=pe, attn_mask=attn_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/flux/sd-scripts/library/flux_models.py", line 452, in attention
x = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 108.00 MiB. GPU 0 has a total capacity of 23.87 GiB of which 63.62 MiB is free. Including non-PyTorch memory, this process has 23.81 GiB memory in use. Of the allocated memory 23.53 GiB is allocated by PyTorch, and 114.40 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
```

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is set in my env

Start with flux_minimal_inference.py around generate_image, do_sample, and denoise, then trace the attention call in library/flux_models.py shown in the traceback. Reproduce inference with both LoRA and negative_prompt enabled, and compare memory use with each option separately; done means the combined case no longer raises CUDA out-of-memory on the reported setup.

Written by the indexing model from the issue text.