Utterance contains multiple speakers

I'm noticing that I'm getting a lot of utterances that contain multiple speakers. This will give me speaker vectors that are calculated from multiple voices. This leads to a lot of bad matches. Any ideas on how I can get better results? Any way to tweak for better utterances between speakers? Any way for the speaker model to detect multiple speakers?

I'm using `python vosk-0.3.18` with `vosk-model-en-us-daanzu-20200905-lgraph` and `vosk-model-spk-0.4`

Thanks!