Using Image to Image VAE Encoder on Flux slow down drastically generation speed and two forward pass triggered

### Expected Behavior

As fast as using EmptyLatentSD3

### Actual Behavior

The title says it all,
Two forwards pass on flux model (one additionnal because of negative_cond ?) instead of one and 2 times slower generation speed. which makes a 4 time slowdown.

### Steps to Reproduce

basic Flux workflow but for image to image on 1024x1024 with vae encoder.

### Debug Logs

```powershell
Flux - Line 1: forward - Variable assignments: 0.000004s
Flux - Line 2: forward - x padding: 0.000118s
Flux - Line 3: forward - img rearrange: 0.000139s
Flux - Line 4: forward - h_len, w_len calculations: 0.000002s
Flux - Line 5: forward - img_ids = torch.zeros(...): 0.000088s
Flux - Lines 6-7: forward - img_ids calculations: 0.000434s
Flux - Line 8: forward - img_ids repeat: 0.000052s
Flux - Line 9: forward - txt_ids = torch.zeros(...): 0.000079s
Flux - Line 1: forward_orig - Dimension check: 0.000003s
Flux - Line 2: forward_orig - img = self.img_in(img): 0.000369s
Flux - Line 3: forward_orig - vec = self.time_in(...): 0.001366s
Flux - Line 4: forward_orig - Conditional guidance embedding: 0.001410s
Flux - Line 5: forward_orig - vec = vec + self.vector_in(y): 0.000772s
Flux - Line 6: forward_orig - txt = self.txt_in(txt): 0.000324s
Flux - Line 7: forward_orig - ids = torch.cat(...): 0.000075s
Flux - Line 8: forward_orig - pe = self.pe_embedder(ids): 0.002902s
Flux - Line 9: forward_orig - Double block 1/19: 0.007156s
Flux - Line 9: forward_orig - Double block 2/19: 0.007398s
Flux - Line 9: forward_orig - Double block 3/19: 0.007165s
Flux - Line 9: forward_orig - Double block 4/19: 0.007517s
Flux - Line 9: forward_orig - Double block 5/19: 0.007938s
Flux - Line 9: forward_orig - Double block 6/19: 0.007994s
Flux - Line 9: forward_orig - Double block 7/19: 0.007971s
Flux - Line 9: forward_orig - Double block 8/19: 0.007767s
Flux - Line 9: forward_orig - Double block 9/19: 0.007668s
Flux - Line 9: forward_orig - Double block 10/19: 0.086609s
Flux - Line 9: forward_orig - Double block 11/19: 0.133773s
Flux - Line 9: forward_orig - Double block 12/19: 0.148316s
Flux - Line 9: forward_orig - Double block 13/19: 0.149675s
Flux - Line 9: forward_orig - Double block 14/19: 0.150106s
Flux - Line 9: forward_orig - Double block 15/19: 0.152614s
Flux - Line 9: forward_orig - Double block 16/19: 0.133540s
Flux - Line 9: forward_orig - Double block 17/19: 0.135913s
Flux - Line 9: forward_orig - Double block 18/19: 0.151441s
Flux - Line 9: forward_orig - Double block 19/19: 0.134774s
Flux - Line 10: forward_orig - ControlNet processing: 0.000003s
Flux - Line 11: forward_orig - img = torch.cat(...): 0.000126s
Flux - Line 12: forward_orig - Single block 1/38: 0.010573s
Flux - Line 12: forward_orig - Single block 2/38: 0.118099s
Flux - Line 12: forward_orig - Single block 3/38: 0.029789s
Flux - Line 12: forward_orig - Single block 4/38: 0.075598s
Flux - Line 12: forward_orig - Single block 5/38: 0.047165s
Flux - Line 12: forward_orig - Single block 6/38: 0.014082s
Flux - Line 12: forward_orig - Single block 7/38: 0.119009s
Flux - Line 12: forward_orig - Single block 8/38: 0.014048s
Flux - Line 12: forward_orig - Single block 9/38: 0.102300s
Flux - Line 12: forward_orig - Single block 10/38: 0.044050s
Flux - Line 12: forward_orig - Single block 11/38: 0.003742s
Flux - Line 12: forward_orig - Single block 12/38: 0.120886s
Flux - Line 12: forward_orig - Single block 13/38: 0.013869s
Flux - Line 12: forward_orig - Single block 14/38: 0.117497s
Flux - Line 12: forward_orig - Single block 15/38: 0.014747s
Flux - Line 12: forward_orig - Single block 16/38: 0.088679s
Flux - Line 12: forward_orig - Single block 17/38: 0.046598s
Flux - Line 12: forward_orig - Single block 18/38: 0.014664s
Flux - Line 12: forward_orig - Single block 19/38: 0.117888s
Flux - Line 12: forward_orig - Single block 20/38: 0.014520s
Flux - Line 12: forward_orig - Single block 21/38: 0.089497s
Flux - Line 12: forward_orig - Single block 22/38: 0.046404s
Flux - Line 12: forward_orig - Single block 23/38: 0.133347s
Flux - Line 12: forward_orig - Single block 24/38: 0.134461s
Flux - Line 12: forward_orig - Single block 25/38: 0.134692s
Flux - Line 12: forward_orig - Single block 26/38: 0.134881s
Flux - Line 12: forward_orig - Single block 27/38: 0.135519s
Flux - Line 12: forward_orig - Single block 28/38: 0.149419s
Flux - Line 12: forward_orig - Single block 29/38: 0.134503s
Flux - Line 12: forward_orig - Single block 30/38: 0.134217s
Flux - Line 12: forward_orig - Single block 31/38: 0.134143s
Flux - Line 12: forward_orig - Single block 32/38: 0.136469s
Flux - Line 12: forward_orig - Single block 33/38: 0.148234s
Flux - Line 12: forward_orig - Single block 34/38: 0.135544s
Flux - Line 12: forward_orig - Single block 35/38: 0.135026s
Flux - Line 12: forward_orig - Single block 36/38: 0.133810s
Flux - Line 12: forward_orig - Single block 37/38: 0.133888s
Flux - Line 12: forward_orig - Single block 38/38: 0.134217s
Flux - Line 13: forward_orig - ControlNet processing: 0.000003s
Flux - Line 14: forward_orig - img slicing: 0.000055s
Flux - Line 15: forward_orig - img = self.final_layer(...): 0.026549s
Flux - Line 10: forward - out = self.forward_orig(...): 4.973931s
Flux - Line 11: forward - out rearrange: 0.000289s
Flux - Line 1: forward - Variable assignments: 0.000006s
Flux - Line 2: forward - x padding: 0.000204s
Flux - Line 3: forward - img rearrange: 0.012521s
Flux - Line 4: forward - h_len, w_len calculations: 0.000003s
Flux - Line 5: forward - img_ids = torch.zeros(...): 0.000159s
Flux - Lines 6-7: forward - img_ids calculations: 0.000759s
Flux - Line 8: forward - img_ids repeat: 0.000096s
Flux - Line 9: forward - txt_ids = torch.zeros(...): 0.000146s
Flux - Line 1: forward_orig - Dimension check: 0.000005s
Flux - Line 2: forward_orig - img = self.img_in(img): 0.000566s
Flux - Line 3: forward_orig - vec = self.time_in(...): 0.021733s
Flux - Line 4: forward_orig - Conditional guidance embedding: 0.103983s
Flux - Line 5: forward_orig - vec = vec + self.vector_in(y): 0.001370s
Flux - Line 6: forward_orig - txt = self.txt_in(txt): 0.042604s
Flux - Line 7: forward_orig - ids = torch.cat(...): 0.000106s
Flux - Line 8: forward_orig - pe = self.pe_embedder(ids): 0.133282s
Flux - Line 9: forward_orig - Double block 1/19: 0.271374s
Flux - Line 9: forward_orig - Double block 2/19: 0.374543s
Flux - Line 9: forward_orig - Double block 3/19: 0.315547s
Flux - Line 9: forward_orig - Double block 4/19: 0.389237s
Flux - Line 9: forward_orig - Double block 5/19: 0.300692s
Flux - Line 9: forward_orig - Double block 6/19: 0.359961s
Flux - Line 9: forward_orig - Double block 7/19: 0.315221s
Flux - Line 9: forward_orig - Double block 8/19: 0.245228s
Flux - Line 9: forward_orig - Double block 9/19: 0.013835s
Flux - Line 9: forward_orig - Double block 10/19: 0.117965s
Flux - Line 9: forward_orig - Double block 11/19: 0.149563s
Flux - Line 9: forward_orig - Double block 12/19: 0.136111s
Flux - Line 9: forward_orig - Double block 13/19: 0.132889s
Flux - Line 9: forward_orig - Double block 14/19: 0.135697s
Flux - Line 9: forward_orig - Double block 15/19: 0.148405s
Flux - Line 9: forward_orig - Double block 16/19: 0.134106s
Flux - Line 9: forward_orig - Double block 17/19: 0.135360s
Flux - Line 9: forward_orig - Double block 18/19: 0.149603s
Flux - Line 9: forward_orig - Double block 19/19: 0.134085s
Flux - Line 10: forward_orig - ControlNet processing: 0.000003s
Flux - Line 11: forward_orig - img = torch.cat(...): 0.000163s
Flux - Line 12: forward_orig - Single block 1/38: 0.004977s
Flux - Line 12: forward_orig - Single block 2/38: 0.124704s
Flux - Line 12: forward_orig - Single block 3/38: 0.014017s
Flux - Line 12: forward_orig - Single block 4/38: 0.089059s
Flux - Line 12: forward_orig - Single block 5/38: 0.047679s
Flux - Line 12: forward_orig - Single block 6/38: 0.005942s
Flux - Line 12: forward_orig - Single block 7/38: 0.125571s
Flux - Line 12: forward_orig - Single block 8/38: 0.013968s
Flux - Line 12: forward_orig - Single block 9/38: 0.104536s
Flux - Line 12: forward_orig - Single block 10/38: 0.028628s
Flux - Line 12: forward_orig - Single block 11/38: 0.014396s
Flux - Line 12: forward_orig - Single block 12/38: 0.106070s
Flux - Line 12: forward_orig - Single block 13/38: 0.015936s
Flux - Line 12: forward_orig - Single block 14/38: 0.131279s
Flux - Line 12: forward_orig - Single block 15/38: 0.014523s
Flux - Line 12: forward_orig - Single block 16/38: 0.074525s
Flux - Line 12: forward_orig - Single block 17/38: 0.046576s
Flux - Line 12: forward_orig - Single block 18/38: 0.014089s
Flux - Line 12: forward_orig - Single block 19/38: 0.118732s
Flux - Line 12: forward_orig - Single block 20/38: 0.014472s
Flux - Line 12: forward_orig - Single block 21/38: 0.089355s
Flux - Line 12: forward_orig - Single block 22/38: 0.045868s
Flux - Line 12: forward_orig - Single block 23/38: 0.134802s
Flux - Line 12: forward_orig - Single block 24/38: 0.134358s
Flux - Line 12: forward_orig - Single block 25/38: 0.148615s
Flux - Line 12: forward_orig - Single block 26/38: 0.135488s
Flux - Line 12: forward_orig - Single block 27/38: 0.134443s
Flux - Line 12: forward_orig - Single block 28/38: 0.135450s
Flux - Line 12: forward_orig - Single block 29/38: 0.133466s
Flux - Line 12: forward_orig - Single block 30/38: 0.134487s
Flux - Line 12: forward_orig - Single block 31/38: 0.135100s
Flux - Line 12: forward_orig - Single block 32/38: 0.149641s
Flux - Line 12: forward_orig - Single block 33/38: 0.134436s
Flux - Line 12: forward_orig - Single block 34/38: 0.133597s
Flux - Line 12: forward_orig - Single block 35/38: 0.134117s
Flux - Line 12: forward_orig - Single block 36/38: 0.134149s
Flux - Line 12: forward_orig - Single block 37/38: 0.135098s
Flux - Line 12: forward_orig - Single block 38/38: 0.135205s
Flux - Line 13: forward_orig - ControlNet processing: 0.000004s
Flux - Line 14: forward_orig - img slicing: 0.000055s
Flux - Line 15: forward_orig - img = self.final_layer(...): 0.025904s
Flux - Line 10: forward - out = self.forward_orig(...): 7.770730s
Flux - Line 11: forward - out rearrange: 0.000216s
Line 3: denoised = model(...): 13.097679s
```

### Other

_No response_