CouncilDataProject / CouncilDataProject/cdp-backend

Speaker classification

Open
#131 18 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
25
Forks
28
PR merge metrics
No merged PRs in 30d

Description

### Feature Description

Backend issue for the relevant [roadmap issue
](https://github.com/CouncilDataProject/cdp-roadmap/issues/5)

Adding speaker classification to CDP transcripts. This could be through a script/class that retroactively attaches the speaker name to a transcript that already has [speaker diarization enabled](https://cloud.google.com/speech-to-text/docs/multiple-voices). Prodigy can be used for annotating the training data.

### Use Case

With speaker classification we can provide transcripts annotated with the speaker. This can be used in many ways such as through a script or github action

### Solution

Very high level idea would be to:
- Use GCP's built-in speaker diarization to separate the speakers. We could also create our own audio classification model. We could also use something like Prodigy to annotate the data, but I believe they have their own diarization/transcription models as well.
- Figure out how to add the classified speaker names to the diarized transcript. I'm not sure if GCP allows you to provide any training data, but from what I could tell they only separate the speakers, but the models don't take in training data to label them.

A bigger picture breakdown of all the major components can be found on the roadmap issue under "Major components".

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.