ml-explore / ml-explore/mlx-examples

Whisper transcribing music sounds

Open
#1,101 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
9k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

I am wondering if people preprocess their audio before sending to whisper? I have sections where there could be music - but whisper seems to just assign different words (like If If If .....) with high probability (am using mlx-community/whisper-large-v3-turbo right now).
Do people 'clean' music out of the clips first? Or have a separate method to determine speaking or not? Or are there specific settings (i don't know about) which would do this for me?

How accurate do people find the timestamps from this version too? I can see people have issues with timestamp accuracy from other whisper large v3 models, but in the limited tests I have done so far it looks pretty decent (but might just be luck or the specific example)

My settings are:
output = mlx_whisper.transcribe(audio_file,path_or_hf_repo="mlx-community/whisper-large-v3-turbo",language="en",fp16=True, word_timestamps=True)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported behavior with mlx_whisper.transcribe using mlx-community/whisper-large-v3-turbo, language="en", fp16=True, and word_timestamps=True. Investigate the music sections, repeated-word output, and timestamp accuracy; the issue is done when the project has a verified explanation or documented guidance for handling these cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.