ml-explore / ml-explore/mlx-examples
Whisper transcribing music sounds
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
I am wondering if people preprocess their audio before sending to whisper? I have sections where there could be music - but whisper seems to just assign different words (like If If If .....) with high probability (am using mlx-community/whisper-large-v3-turbo right now).
Do people 'clean' music out of the clips first? Or have a separate method to determine speaking or not? Or are there specific settings (i don't know about) which would do this for me?
How accurate do people find the timestamps from this version too? I can see people have issues with timestamp accuracy from other whisper large v3 models, but in the limited tests I have done so far it looks pretty decent (but might just be luck or the specific example)
My settings are:
output = mlx_whisper.transcribe(audio_file,path_or_hf_repo="mlx-community/whisper-large-v3-turbo",language="en",fp16=True, word_timestamps=True)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported behavior with mlx_whisper.transcribe using mlx-community/whisper-large-v3-turbo, language="en", fp16=True, and word_timestamps=True. Investigate the music sections, repeated-word output, and timestamp accuracy; the issue is done when the project has a verified explanation or documented guidance for handling these cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100