Can it be used to improve speaker-diarization performance for English?

Hi,

I am working on a project where I want to create speaker-aware transcripts from audios/videos, preferably using open-source solutions. I have tried so many approaches but nothing seems to work good enough out of the box.

I have tried:

1. whisperX: https://github.com/m-bain/whisperX (uses pyannote)
2. whisper-diarization: https://github.com/MahmoudAshraf97/whisper-diarization (uses Nemo)
3. AWS Transcribe
4. AssemblyAI API
5. Picovoice API

I read this in the fine-tuning google colab notebook: "The segmentation model has been trained on a combination of datasets containing mostly English and Mandarin languages. As a consequence, the performance of the speaker diarization pipeline may decrease when confronted with out-of-distribution data, such as Spanish or Japanese."

Does that mean fine-tuning won't help much in case of english speakers? My doubt is if pyannote pretrained models are already trained in English and Mandarin, how will fine-tuning using some more english data help with diarization accuracy!

I am yet to find a speaker-diarization solution that is accurate enough for english audios and am looking for suggestions for improvements.

Start with the fine-tuning Google Colab notebook and the issue's discussion of pyannote pretrained models, English data, and diarization accuracy. Establish whether a concrete improvement is intended, then define an evaluation using English audio; done requires an agreed scope and measurable accuracy result.