Multi-speaker training failing

Hello,

I just trained on two speakers at the same time.
The filelist looks like this:

/home/ubuntu/RVC-beta-v2-0528/logs/merged/0_gt_wavs/0_4_48.wav|/home/ubuntu/RVC-beta-v2-0528/logs/merged/3_feature768/0_4_48.npy|/home/ubuntu/RVC-beta-v2-0528/logs/merged/2a_f0/0_4_48.wav.npy|/home/ubuntu/RVC-beta-v2-0528/logs/merged/2b-f0nsf/0_4_48.wav.npy|0
/home/ubuntu/RVC-beta-v2-0528/logs/merged/0_gt_wavs/1_2_6.wav|/home/ubuntu/RVC-beta-v2-0528/logs/merged/3_feature768/1_2_6.npy|/home/ubuntu/RVC-beta-v2-0528/logs/merged/2a_f0/1_2_6.wav.npy|/home/ubuntu/RVC-beta-v2-0528/logs/merged/2b-f0nsf/1_2_6.wav.npy|1
...

I have 184 samples of the first speaker, and 169 of the second, so fairly balanced.
When running Feature Extraction, I used Crepe with a lower hop for speaker 0 than for speaker 1.

I have double-checked that the audio files and the speaker id go in correctly into the forward pass of the model.
The speaker embeddings for 0 and 1 also come out different after training is finished, so something worked.

However, when inferencing on new audio, the output when using speaker ID 0 and speaker ID 1 both sound identical (and mostly like speaker 0, but sometimes they both sound like speaker 1 - for example when inferencing on training data for speaker 1).
They also look almost identical - only tiny differences that are not audible to my ear when comparing them in Audacity.
This is true when comparing the trained model after 1 epoch, 125 epochs, or 300 epochs (and in between).
It's almost like something went wrong with the speaker separation/disentangling.

When training the two speakers by themselves (one model each), the difference is very noticeable.

Any tips on where I might have gone wrong in training on multiple speakers?

Many thanks in advance!